{"id":"514a137d-41cf-4e9e-b4f6-9a693d7c197e","arxiv_id":"2505.12761","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Adding Crossformer-style router attention to Time-LLM patch embeddings improves average MSE on Weather and a 10-feature Traffic subset, while degrading ETTh2, ETTm1, and ETTm2.","lead":"This paper adds a cross-variate router attention module to the patch embedding step of Time-LLM, a channel independent forecasting model, and tests the modified model against the original on seven datasets. The approach shows small MSE gains on correlated benchmarks like Weather and Traffic, but it loses on three of seven datasets and the experiments lack error bars and use modified datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported CVPE gains are measured against a modified Time-LLM baseline (GPT-2, T=256) and on correlation-selected Traffic/ECL subsets, without error bars; the 4.6% Weather and 6.7% Traffic improvements may not persist under the original configuration or on full datasets.","rationale":"I read the paper as making a comparative claim: inserting CVPE into Time-LLM improves forecasting on datasets with rich cross-variate dependencies. The evidence for this claim is Table 1 and Appendix C. The most insecure link is the evaluation protocol. The paper's own limitations section acknowledges that GPU constraints forced T=256, GPT-2, and modified Traffic/ECL datasets. These choices alter both the baseline and the test distribution, and the top-10 correlated features are selected by the very criterion the module is designed to exploit, creating a risk of selection bias. The absence of error bars compounds the problem: with single-run averages over four horizons, a 4-7% difference cannot be distinguished from noise. The internal inconsistency between the abstract's unqualified 'outperforms the original baseline' and Table 1's losses on ETTh2, ETTm1, and ETTm2 further weakens confidence, though the qualified claim in the strongest_claim is the operative one. I agree with the reader's weakest_assumption and would not change the CONDITIONAL verdict: a full-dataset, repeated-seed replication is the decisive test. Credit is due for releasing code and for a minimal architectural change that is clearly described; no issue of internal logic in the CVPE equations would change the verdict without the empirical evidence. The concern is about the strength and generalizability of the comparative evidence, not about the validity of the model construction or the honesty of the reporting.","tokens_in":8616,"tokens_out":7899,"duration_ms":83033,"concrete_test":"Recompute the comparison on the full Traffic and ECL datasets (all original channels, no feature selection) using the same GPT-2/T=256 configuration, running each model over at least five random seeds and reporting per-horizon mean and standard deviation of MSE. If the CVPE advantage on either full dataset disappears or falls within one standard deviation, the reported gains are not robust evidence for the central claim. This single check also supplies the missing error bars needed to assess the four-horizon averages in Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on a fair and representative comparison between CVPE-enhanced Time-LLM and the original Time-LLM. In Table 1, 'TIME-LLM (Original)' is not the original model from Jin et al. (2024): Section 4.1 and Appendix B.3 state that Llama-7B was replaced with GPT-2 and the context window T was reduced from 512 to 256. Both arms share this altered baseline, so the relative comparison is internally consistent, but the paper's broader claim of outperforming 'the original baseline' is never tested against the original configuration. More importantly, Traffic and ECL are reduced to the ten features with the highest Pearson correlation (Section 4.1, Appendix B.1). This post hoc selection creates precisely the regime where a cross-variate module has the most opportunity to help, and the full-dataset case is not evaluated. The reported gains are averages over four horizons, apparently from a single run, and no standard deviations or significance tests are provided. Given that the method loses on ETTh2 and ETTm2 by up to 5.2%, it is plausible that the advantage on the selected subsets is within run-to-run noise or reverses under the original T=512 or Llama-7B setting. The missing full-dataset and repeated-run evidence is load-bearing because the central claim is a comparative performance claim, and the current setup is the one most favorable to CVPE.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Cross-Variate Patch Embeddings (CVPE), a lightweight module that injects cross-variate information into channel-independent (CI) forecasting models by modifying only the patch embedding step. The module adds a learnable positional encoding and a router-attention block, inspired by Crossformer, to the vanilla patch embedding used in Time-LLM. The authors integrate CVPE into Time-LLM and compare against a re-implemented Time-LLM baseline on seven benchmarks (four ETT datasets, Weather, and correlation-reduced Traffic and ECL subsets) over four forecast horizons, reporting MSE and MAE. The main claim is that CVPE improves forecasting performance on datasets with rich cross-variate dependencies, with up to 4.6% average MSE gain on Weather and 6.7% on the modified Traffic subset, while acknowledging performance losses on ETTh2 and ETTm2.","tokens_in":8975,"tokens_out":4859,"duration_ms":49003,"significance":"If the empirical claims held, CVPE would be a simple and practical plug-in for augmenting CI models with cross-variate context, requiring no changes to the backbone and adding modest complexity. The paper's strengths are its minimal architectural intervention, clear attribution of the router-attention idea to Crossformer, and the public release of code. However, the current evidence is not sufficient to establish the central claim: the positive results are concentrated on datasets that the authors have modified by selecting the ten most correlated features, the baseline is not the original Time-LLM configuration, and all comparisons appear to come from a single run without error bars. The negative results on three of the seven benchmarks (up to 5.2% degradation) further complicate the advertised advantage. The contribution is potentially useful, but the empirical case needs substantially more support before the paper's conclusions can be accepted.","major_comments":[{"comment":"The blanket statement that the CVPE-enhanced model 'outperforms the original baseline' is not supported by the full table. Averaged over the four horizons, CVPE is worse on ETTh2 (MSE 0.385 vs 0.366, about 5.2% degradation), ETTm1 (0.390 vs 0.382), and ETTm2 (0.289 vs 0.275). The positive gains are concentrated in Weather and modified Traffic, with tiny gains on ETTh1 and modified ECL. The abstract and introduction should be reworded to describe selective gains rather than overall improvement, or the failures should be analyzed as part of the main claim.","section":"§4.1, Table 1"},{"comment":"The Traffic and ECL datasets are reduced to the ten features with the highest Pearson correlation. This data-dependent selection creates a regime that is most favorable to a cross-variate module, and results on the full Traffic and ECL datasets are not reported. Consequently, the claim that CVPE improves 'key datasets with rich cross-variate dependencies' is not established for the actual benchmarks practitioners use. The authors should evaluate on the full datasets, or explicitly justify that the selected subsets are representative of the cross-variate structure in the original data.","section":"§4.1, Eq. (2); Appendix B.1"},{"comment":"The baseline labeled 'TIME-LLM (Original)' in Table 1 is not the original Time-LLM: the LLM backbone is changed from Llama-7B to GPT-2 and the context window T is reduced from 512 to 256. The comparison is internally consistent because both arms share this altered backbone, but the paper's wording 'outperforms the original baseline' is misleading and is never tested against the original configuration. The authors should either report results under the original Llama-7B/T=512 setting (even for a subset of datasets) or clearly restrict all claims to the modified re-implementation.","section":"§4.1, Appendix B.3"},{"comment":"No error bars, standard deviations, or significance tests are reported. Many of the observed differences are small (e.g., ETTh1 MSE 0.445 vs 0.453; ECL Modified 0.191 vs 0.192), and the averages are apparently computed from a single run. Given the mixed results across datasets, repeated runs with reported mean and standard deviation (or a significance test) are necessary to support the comparative performance claim.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"The Pearson correlation formula has a typo: the denominator repeats the sum of squared deviations for x in both factors; the second factor should involve (y_i - \\bar{y})^2.","section":"Eq. (2)"},{"comment":"The router vectors are introduced as 'for each time step j', but the notation R \\in \\mathbb{R}^{N\\times c\\times d_m} has no time dimension and j never appears in Eq. (1). The description should say 'for each variate' or clarify the role of j.","section":"§3.2"},{"comment":"The configuration table lists 'Electricity' and 'Traffic', but the experiments use the modified ten-feature subsets. The labels should be consistent (e.g., 'ECL (Modified)' and 'Traffic (Modified)') throughout.","section":"Table 3 and Appendix B.1"},{"comment":"The abstract says 'seven real-world datasets,' but two of them are correlation-reduced subsets of Traffic and ECL, not the original benchmarks. This should be clarified.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short empirical contribution whose central claim is currently overstated relative to the evidence. The main risks are the data-selected Traffic/ECL subsets and the modified Time-LLM baseline; both are fixable with additional experiments and more careful wording. The code release is a positive factor. I would not support acceptance without full-dataset results or a clear scope restriction, and without multiple-seed statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one-line take: this is a modest, honest incremental paper that takes a known mechanism (Crossformer's router attention) and inserts it into the patch embedding of a channel-independent model (Time-LLM). That specific combination is new, and the authors are transparent about where it fails. But the headline \"outperforms\" outruns the evidence, mainly because the baseline is not the original Time-LLM and two datasets were pruned to the ten most correlated channels.\n\nWhat it does well: the idea is clean and cheap. Add a learnable positional encoding and a router attention block at the patch embedding level, keep the rest of Time-LLM untouched, and you inject cross-variate information without converting the model to a full channel-dependent one. The complexity stays O(NP), which is genuinely lightweight. The authors also report per-horizon results and admit the losses on ETTh2 and ETTm2. Code is public.\n\nThe soft spots are real but not fatal for an exploratory paper. First, both arms of Table 1 use GPT-2 and context length 256, not Llama-7B and 512 as in the original Time-LLM. So the comparison is internally consistent but does not test \"outperforms the original baseline\" in the published sense. Second, Traffic and ECL are reduced to the top ten features by Pearson correlation. That selection is post hoc and picks exactly the regime where cross-variate reasoning should help; we don't get to see whether the benefit survives on the full datasets. Third, the numbers are single-run averages over four horizons with no error bars, and the method is worse on ETTh2/ETTm2 by up to 5.2%. So the 4.6% Weather and 6.7% modified-Traffic gains could be within run-to-run noise.\n\nI don't think the stress-test note overreaches: the external validity of the central claim is genuinely in question. The authors themselves flag the memory constraints and call for full-dataset evaluation, which is honest but does not fix the current evidence.\n\nWho should read it: anyone working on CI vs CD trade-offs in time series forecasting, or on lightweight ways to reintroduce cross-variate dependencies. For a referee, I'd send it out, because the idea is plausible and the execution is clear; but I'd ask for the full datasets, repeated runs, standard deviations, and a version of the comparison against the real original Time-LLM. As is, it's a useful incremental data point, not a mature claim.\n\nI'd bring it to a reading group as a case study in evaluation choices. But I wouldn't cite it as support for a performance claim yet.","headline":"A clean, honest incremental idea whose central empirical claim is undercut by a non-original baseline and correlation-pruned datasets; worth reviewing, but not at face value.","tokens_in":9438,"tokens_out":2378,"would_cite":false,"duration_ms":24029,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a channel-independent time series forecaster can absorb cross-variate information by swapping only its patch-embedding layer for a version with a learnable position encoding and a router-attention block, improving…","keywords":["channel independence","cross-variate dependencies","patch embedding","router attention","long-term time series forecasting","Time-LLM","transformer forecasting","multivariate forecasting"],"falsifier":"Re-run the comparison on the full Traffic and ECL datasets with the original Llama-7B backbone and a 512-step context: if the reported 6.7% and 4.6% average MSE gains do not reproduce or reverse, the benefit is an artifact of the reduced datasets or the weaker baseline.","tokens_in":8419,"feed_emoji":"📈","tokens_out":6676,"duration_ms":58539,"temperature":0.7,"pith_summary":"This paper argues that channel-independent (CI) forecasting models, which treat each variable as a separate univariate problem, can be given cross-variate awareness without giving up their robustness or redesigning the whole model. The proposed Cross-Variate Patch Embedding (CVPE) replaces the vanilla patch embedding with one that adds a learnable positional encoding and a lightweight router-attention block, so each patch carries information gathered from all variables before the frozen LLM sees it. Plugging CVPE into Time-LLM lowers average MSE by 4.6% on Weather and 6.7% on a ten-channel subset of Traffic, with no meaningful loss on ETTh1 or the reduced ECL set. On ETTh2 and ETTm2, where channel correlations are weak, the module loses up to 5.2% on average MSE, so the benefit is conditional on how much inter-variable signal actually exists. If the claim holds, CI models can be upgraded incrementally rather than replaced by fully channel-dependent architectures.","feed_headline":"Cross-variate patch embedding lifts Time-LLM forecasts","feed_subtitle":"A small router-attention block added to patch embeddings improves average MSE by 4.6% on Weather and 6.7% on reduced Traffic.","key_machinery":"The load-bearing object is the Cross-Variate Patch Embedding module: after linear projection of each channel's patches, the model adds a learnable position encoding $W_P \\in \\mathbb{R}^{P \\times d_m}$ to the stacked embeddings $X_P \\in \\mathbb{R}^{N \\times P \\times d_m}$, then runs two multi-head attention steps with $c$ learnable router vectors $R \\in \\mathbb{R}^{N \\times c \\times d_m}$. The first step aggregates all variates into a compact router representation; the second redistributes that representation back to the patches, followed by a LayerNorm with skip connection and an MLP. Its stated complexity is $O(NP)$, which is what keeps the module 'lightweight'.","core_discovery":"The central claim is that injecting cross-variate context at the patch-embedding stage is enough to let a channel-independent model like Time-LLM exploit inter-variable relationships. The authors show that adding a learnable position encoding plus a router-attention mechanism to the patch embeddings, and nothing else, yields consistent MSE and MAE improvements on Weather and a ten-feature Traffic subset while leaving weakly correlated ETT datasets essentially unchanged or slightly worse. This is presented as evidence that the channel-independent design, not the lack of a cross-channel mechanism, is the main thing separating CI models from richer multivariate models.","pith_inferences":["A natural extension would be to graft CVPE onto other patch-based CI forecasters, such as PatchTST-style encoders; if the router-attention transfer is the active ingredient, similar gains should appear there.","The evaluation's dependence on ten-feature subsets of Traffic and ECL means the headline gains could be partly an artifact of selecting highly correlated channels; testing on the full datasets would separate the module's effect from the selection effect.","The overfitting on ETTh2 and ETTm2 suggests a promising variant that restricts router attention to a subset of channels per patch, trading a bit of cross-variate capacity for lower noise.","If the $O(NP)$ complexity holds at larger channel counts, CVPE could offer a cheap way to keep CI models competitive with fully channel-dependent models without paying their full attention cost."],"forward_implications":["Channel-independent forecasters can be retrofitted with cross-channel information through a small, self-contained module; the rest of the pipeline, including the frozen LLM, stays untouched.","On datasets whose variables are strongly correlated, average MSE drops by 4.6% on Weather and 6.7% on the reduced Traffic set, and the improvement appears across all four tested horizons.","On weakly correlated datasets, the module does no consistent harm to ETTh1 or the reduced ECL set, suggesting the extra capacity can be absorbed without degrading temporal modeling.","On ETTh2 and ETTm2, gains reverse to losses of up to 5.2%, so the module's usefulness depends on the strength of real cross-variate signal.","Because CVPE is inserted before patches are split back into per-channel streams, the cross-variate information survives the channel-independent reprogramming and LLM layers."],"supporting_citations":[{"why":"Defines the Time-LLM baseline whose patch-embedding step CVPE replaces and whose experimental configurations the paper follows.","marker":"(Jin et al., 2024)"},{"why":"Supplies the router-attention design that CVPE adapts for cross-variate aggregation.","marker":"(Zhang & Yan, 2023)"},{"why":"Establishes the channel-independent patch-embedding paradigm that CVPE extends.","marker":"(Nie et al., 2023)"},{"why":"Frames the capacity-robustness trade-off between channel-dependent and channel-independent modeling that motivates adding a lightweight CD module.","marker":"(Han et al., 2023)"},{"why":"Provides the benchmark datasets (Weather, Traffic, ECL) and their statistics.","marker":"(Wu et al., 2023)"},{"why":"GPT-2 is the LLM backbone used in this paper's re-implementation of the baseline.","marker":"(Radford et al., 2019)"},{"why":"Llama-7B is the original Time-LLM backbone that memory constraints replaced.","marker":"(Touvron et al., 2023)"},{"why":"Supports the claim that fully channel-dependent models are vulnerable to overfitting, motivating a channel-independent hybrid.","marker":"(Qiu et al., 2025)"}],"fun_headline_variants":["Cross-variate patch context strengthens CI forecasting","Add cross-variate info at patch embedding for CI gains","Tiny router-attention helps CI models capture variable links","Simple patch tweak lets CI models see cross-variate signals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gains depend on comparing against a Time-LLM re-implementation that uses GPT-2 with a 256-step context, and on Traffic and ECL datasets reduced to their ten most correlated channels; if these do not match how practitioners actually run Time-LLM, the measured improvements could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Cross-variate patch context strengthens CI forecasting","Add cross-variate info at patch embedding for CI gains","Tiny router-attention helps CI models capture variable links","Simple patch tweak lets CI models see cross-variate signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000319,"raw_usage":{"total_tokens":1758,"prompt_tokens":864,"completion_tokens":894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":829}},"tokens_in":480,"tokens_out":894,"duration_ms":9294,"temperature":1.0,"reasoning_tokens":829,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:25:55.655409+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on the full Traffic and ECL datasets with the original Llama-7B backbone and a 512-step context: if the reported 6.7% and 4.6% average MSE gains do not reproduce or reverse, the benefit is an artifact of the reduced datasets or the weaker baseline.","supporting_citations":[{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"GPT-2 is the LLM backbone used in this paper's re-implementation of the baseline."}],"review_version":1}