{"id":"a400eac8-7d58-46bd-8831-0c01a191ee46","arxiv_id":"2608.08119","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A unified, configuration-driven toolbox benchmarks time-series dataset similarity metrics and shows that no single metric consistently predicts downstream task performance.","lead":"This paper presents a software toolbox for measuring how similar two time-series datasets are, and for checking whether those similarity scores predict how well models transfer between datasets. The toolbox standardizes evaluation so researchers can compare different similarity metrics under the same conditions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pooled Pearson correlations in Table 2 are confounded by target-dataset difficulty: raw MSE is not normalized per target, so the 'no consistent metric' claim may be an artifact of spurious correlations with target volatility.","rationale":"The paper is primarily a software/benchmarking contribution, and the toolbox architecture is plausible and useful. The empirical demonstration is the weakest link, and the reader correctly identified the Pearson-correlation-based evaluation as the load-bearing assumption. My concern sharpens this: the pooled correlation is not merely under-powered or lacking error bars; it is potentially biased by target-difficulty confounding, which can manufacture spurious positive correlations and directly undermine the 'no consistent metric' conclusion. This warrants a concrete re-analysis before the empirical claim is accepted. Since the reader already recommended CONDITIONAL acceptance pending stronger evaluation, my analysis supports keeping that verdict, hence UNCHANGED. I would not reject the paper, because the toolbox itself may be valid and the evaluation can be fixed with target-wise normalization. I partially agree with the reader's weakest assumption: the reader pointed to validity/comprehensiveness of the Pearson correlation, and I identify a specific validity threat that was not explicitly articulated. The proposed test is straightforward and uses existing outputs, so it is a feasible condition for acceptance.","tokens_in":8610,"tokens_out":5672,"duration_ms":60404,"concrete_test":"Recompute the correlations in Table 2 after target-standardizing the downstream losses: for each target dataset, z-score the MSE (or membership error) across sources, then compute the pooled correlation with distances; also report the average within-target Spearman correlation. If the ranking of metrics changes materially, or if all correlations collapse toward zero, the original claim is unsupported. This test requires no new data—only re-analysis of the existing task-performance matrices.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The empirical conclusion that no similarity metric consistently aligns with downstream performance (Section 5, Tables 2–3) rests on Pearson correlations between pairwise dataset-distance scores and raw downstream losses. For forecasting, E_{X,Y} is the MSE of a source-trained model evaluated on the target dataset Y (Section 4.4). This MSE is not standardized per target. Different target datasets have very different intrinsic forecastability; the z-score normalization of input windows (Section 5) does not equalize target-specific irreducible error. When all source–target pairs are pooled for correlation, the dominant variance can be at the target level: hard targets yield high MSE for every source. Any similarity metric whose distance is correlated with target volatility—for example, WSD explicitly depends on sample covariance, and MMD/OT capture distributional spread—will then show a spuriously positive pooled correlation even if, within each target, the metric does not rank sources by transfer quality. The same confounding applies to the classification task, where one-class membership rates are low for targets that are globally far from all sources. The paper provides no within-target correlations, no baseline such as random source ranking, and no control for target difficulty. Thus the observed pattern, and the key claim that no metric is consistently best, may reflect pooling artifacts rather than genuine properties of the similarity methods.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TSDS-Toolbox, a configuration-driven Python framework for computing pairwise time-series dataset similarity using dataset-level metrics (WSD, MMD, OT, MAD) and reducer-based series-level metrics (DBA/PCA reducers with DTW, ED, LCSS distances). It also provides downstream evaluation pipelines for classification and forecasting, and uses Pearson correlation between similarity distances and task losses to assess which metrics align with transfer performance. Experiments on 25 GluonTS datasets indicate that no single similarity metric consistently outperforms others across tasks, and that reducer choice matters. The code is publicly available.","tokens_in":8923,"tokens_out":4467,"duration_ms":45082,"significance":"If the toolbox is robust and the empirical findings hold, it would provide a valuable standardized benchmarking resource for time-series dataset similarity, addressing a real fragmentation in the area. The paper ships code and a configurable pipeline, which is a strength. The empirical claim about task-dependence of similarity metrics is potentially useful but is currently under-supported by the presented analysis.","major_comments":[{"comment":"The pooled Pearson correlations between similarity distances and raw downstream MSE/classification losses are likely confounded by target-dataset difficulty. Because E_{X,Y} is the raw MSE on target Y without per-target standardization, hard targets contribute high losses for every source; if a similarity metric correlates with target volatility (e.g., WSD depends on sample covariance), the pooled correlation can be positive even when the metric does not rank sources correctly within any target. The paper reports no within-target correlations, no per-target normalization, and no random-ranking baseline. This directly affects the main empirical conclusion that no similarity metric is consistently best, so the analysis must be re-done with target-level controls.","section":"Section 5, Tables 2 and 3"},{"comment":"The experimental protocol omits values for several load-bearing hyperparameters: the k in kNN membership, the quantile threshold tau, the MMD kernel bandwidth, MAD alpha/beta, DBA iteration count, Sinkhorn regularization (if used), and the fine-tuning hyperparameters for Lag-Llama and Time-MoE. Tables 2 and 3 are the only empirical evidence, and without these settings the results are not reproducible from the paper alone. The toolbox's reproducibility claim requires at least a hyperparameter table and seed information.","section":"Section 4.4 and 5"},{"comment":"The experiments use exactly one dataset pool (25 GluonTS datasets) and one sampling configuration (100 windows of length 100). The abstract and introduction claim 'diverse experimental settings,' but no variation in corpus, sampling, or normalization is reported. The claim that the findings generalize across settings is therefore unsupported. Please either add experiments with different pools/settings or temper the claim accordingly.","section":"Section 5"},{"comment":"The correlation values are point estimates derived from a single run; no confidence intervals, p-values, or repeated seeds are reported. Consequently, the paper cannot support its comparison of metrics (e.g., MAD vs WSD in OOD-TMoE) or its claim about consistency across tasks. Please provide error bars or significance testing.","section":"Section 5, Tables 2 and 3"}],"minor_comments":[{"comment":"The copyright block reads 'Conference acronym ’XX, Woodstock, NY ©2018'; the year should match the intended publication venue.","section":"First page"},{"comment":"In the WSD formula, (Σ_X Σ_Y)^{1/2} is the matrix square root; please state this explicitly to avoid ambiguity.","section":"Section 4.3"},{"comment":"The 'Equal Length' column shows ✗ for MAD, but the text says MAD supports different lengths; please clarify the intended meaning of that column entry.","section":"Table 1"},{"comment":"Reference [7] is co-authored by two of the paper's authors; please add a disclosure or footnote to make the self-citation transparent.","section":"Reference [7]"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical claim is the main weakness; the target-difficulty confound in the pooled correlations must be addressed before the results can be considered reliable. Also, WSD is drawn from a paper co-authored by two of the current authors, and the evaluation shows WSD performing strongly; please ensure this is not perceived as a conflict of interest. The toolbox itself seems useful and the code availability is commendable, but the validation is too thin for the strong conclusion in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the toolbox is a real contribution, the evaluation is not. The paper's headline finding—that no similarity metric consistently tracks downstream loss—rests on pooled Pearson correlations that are likely confounded by target-dataset difficulty. That doesn't sink the toolbox, but it should sink the current version of the empirical claim.\n\nWhat's actually new is the integration. WSD, MMD, OT, MAD, DBA/PCA reducers, and DTW/ED/LCSS distances all exist elsewhere. Putting them in one config-driven YAML pipeline with evaluation hooks for classification (one-class kNN membership) and forecasting (Lag-Llama, Time-MoE fine-tuning) is useful. The modular design and public code address a real fragmentation problem. I'd want to try it.\n\nThe soft spot is the validation. Tables 2 and 3 report Pearson correlations pooled over source-target pairs, with raw MSE (and membership rates) as the task loss. Raw MSE is not normalized per target. A hard target yields high MSE for every source, and any metric whose distance tracks target volatility—WSD by construction, MMD/OT by distributional spread—will show a spurious positive pooled correlation. The paper has no within-target correlations, no random-source baseline, and no control for target difficulty. So \"no metric is consistently best\" may be an artifact, and the specific high correlations for WSD/MAD in Time-MoE OOD could reflect target-level variance rather than genuine ranking of sources. There are also no confidence intervals, seeds, or repeated runs, only one pool of 25 datasets, and free hyperparameters like MMD bandwidth and kNN k are left without sensitivity analysis.\n\nThe self-citation of WSD (ref [7], two co-authors) is not itself a problem, but given the confound, the WSD result deserves scrutiny, not a pass.\n\nBottom line: this is a software paper with a promising artifact and a weak empirical section. A good referee would ask for per-target analysis, a random baseline, and reproducibility artifacts (configs, seeds, error bars). I'd send it to review with that expectation.","headline":"A genuinely useful integration toolbox whose headline empirical claim about similarity metrics is undermined by a pooling confound.","tokens_in":9433,"tokens_out":2782,"would_cite":false,"duration_ms":28723,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TSDS-Toolbox standardizes time-series dataset similarity benchmarking, and its experiments indicate that no single similarity metric consistently predicts downstream task performance.","keywords":["time-series dataset similarity","benchmarking framework","similarity metrics","time-series foundation models","source dataset selection","transfer learning","forecasting","dynamic time warping"],"falsifier":"Repeat the comparison on a much larger and more diverse set of public time-series datasets, or measure utility by full end-task accuracy after fine-tuning instead of Pearson correlation with two proxy losses; if one similarity metric then ranks first across classification, out-of-distribution forecasting, and transfer-learning forecasting, the paper's central claim would be falsified.","tokens_in":8435,"feed_emoji":"🧰","tokens_out":11078,"duration_ms":102724,"temperature":0.7,"pith_summary":"This paper presents TSDS-Toolbox, a configuration-driven software framework for measuring and benchmarking similarity between time-series datasets. It brings dataset-level metrics (including Wasserstein distance, maximum mean discrepancy, optimal transport, and match-and-deform) and reducer-based series-level metrics into one pipeline, then evaluates how well each metric predicts performance on downstream forecasting and classification tasks. The paper's central empirical claim is that no single similarity method consistently aligns with downstream task performance; the best metric depends on the task and on the forecasting model used. If this claim is right, the toolbox gives researchers a practical, reproducible way to choose source datasets and to compare similarity metrics for transfer learning instead of relying on a single default measure.","feed_headline":"No single metric wins across time-series similarity tasks","feed_subtitle":"On 25 datasets, no similarity metric consistently tracks downstream loss across forecasting and classification tasks.","key_machinery":"The central mechanism is a four-layer pipeline. A Data Layer converts heterogeneous time-series inputs into a common representation with consistent preprocessing; a Similarity Layer computes pairwise distance matrices for both dataset-level metrics and series-level metrics, where series-level metrics first apply a reducer (DTW Barycenter Averaging or PCA) to map each dataset to a representative sequence and then apply a series distance (DTW, Euclidean distance, or longest common subsequence). An Evaluation Layer measures whether those distances carry information about downstream performance, using a one-class k-NN membership rate for classification and fine-tuned foundation-model forecast MSE for forecasting, in both out-of-distribution and transfer-learning settings. An Analysis Layer then computes Pearson correlations between the distance matrices and the task losses, along with heatmap and network-graph visualizations, which is the step that makes different similarity methods comparable on equal footing.","core_discovery":"The central discovery is that time-series dataset similarity is task-dependent, and the toolbox is designed to make that dependency visible and measurable. Across the evaluated datasets and models, Pearson correlations between dataset-distance scores and downstream losses change substantially from one setting to another: dataset-level metrics such as match-and-deform and Wasserstein distance align most strongly with out-of-distribution forecasting loss when paired with one forecasting foundation model, while optimal transport aligns best with the one-class classification proxy, and no metric is consistently top-ranked across all columns. The experiments also show that DBA-based reduction generally tracks downstream performance better than PCA-based reduction, and that LCSS and MMD show weak alignment throughout. The paper concludes that dataset similarity should be assessed per task and that a standardized benchmarking environment is the appropriate basis for that assessment.","pith_inferences":["A natural extension of the toolbox's correlation tables is to learn a meta-model that, given a target task and model, predicts which similarity metric will align best with downstream loss; the paper stops at reporting the rankings, but its modular design makes this straightforward.","Since the empirical conclusion is based on only 25 datasets and two forecasting models, the same protocol should be run on a larger and more heterogeneous collection; the observed task-dependence may turn out to be an artifact of the small sample, or it may sharpen into a more concrete rule about which metric fits which task.","A user could run the toolbox offline on their own domain's datasets to pick a similarity metric before doing transfer learning, effectively turning the paper's comparative results into a per-domain recommendation procedure that the paper does not itself propose.","One could extend the evaluation layer to regression, anomaly detection, or generation tasks; if the task-dependence result holds there, dataset similarity would be even more clearly a per-task design choice rather than a property of the data alone."],"forward_implications":["Researchers can now compare any newly proposed dataset-similarity metric against existing ones under identical data preprocessing, reducers, and downstream evaluation, making benchmark results directly comparable across papers.","In forecasting out-of-distribution settings, dataset-level metrics (match-and-deform and Wasserstein distance) are reasonable first choices for source-dataset selection, based on their higher correlations in the experiments.","DBA reducers are generally preferable to PCA reducers for downstream alignment in these experiments, suggesting that averaging-based reduction retains more task-relevant information than projection-based reduction.","LCSS and MMD show weak alignment with downstream performance, so users should be cautious about using them as proxies for transferability.","The task-dependence result implies that source-dataset selection pipelines should evaluate candidate similarity metrics on the actual downstream task rather than trusting one metric a priori."],"supporting_citations":[{"why":"Supplies the dataset collection used for all evaluation experiments.","marker":"[1]"},{"why":"Defines the optimal-transport dataset distance implemented in the Similarity Layer.","marker":"[2]"},{"why":"Defines the Wasserstein dataset-similarity metric and motivates its connection to out-of-distribution and transfer loss.","marker":"[7]"},{"why":"Defines the maximum mean discrepancy estimator used as a dataset-level metric.","marker":"[16]"},{"why":"Defines the match-and-deform algorithm that combines optimal transport with DTW alignment.","marker":"[26]"},{"why":"Defines DTW Barycenter Averaging, one of the two reducers used for series-level metrics.","marker":"[30]"},{"why":"Defines dynamic time warping, a series-level distance used in reducer-based evaluation.","marker":"[33]"},{"why":"Defines Pearson correlation, the statistic used to compare distance matrices with task losses.","marker":"[28]"},{"why":"Supplies one of the forecasting foundation models used in out-of-distribution evaluation.","marker":"[32]"},{"why":"Supplies the other forecasting foundation model used in out-of-distribution and transfer-learning evaluation.","marker":"[34]"}],"fun_headline_variants":["No single metric dominates time-series similarity","Similarity metric choice hinges on the time-series task","Benchmarking reveals task-dependent similarity metrics","Time-series similarity: one size does not fit all","Best similarity metric varies with downstream task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's empirical conclusion rests on assuming that the correlation between a similarity score and two chosen performance measures (a classification membership rate and forecast error) on 25 sampled datasets is a complete measure of a metric's usefulness; if that assumption fails, the claim that no metric is consistently better is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["No single metric dominates time-series similarity","Similarity metric choice hinges on the time-series task","Benchmarking reveals task-dependent similarity metrics","Time-series similarity: one size does not fit all","Best similarity metric varies with downstream task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1154,"prompt_tokens":853,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":233}},"tokens_in":469,"tokens_out":301,"duration_ms":3780,"temperature":1.0,"reasoning_tokens":233,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:22:26.778923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the comparison on a much larger and more diverse set of public time-series datasets, or measure utility by full end-task accuracy after fine-tuning instead of Pearson correlation with two proxy losses; if one similarity metric then ranks first across classification, out-of-distribution forecasting, and transfer-learning forecasting, the paper's central claim would be falsified.","supporting_citations":[{"cited_title":"Gluonts: Probabilistic and neural time series modeling in python.Journal of Machine Learning Research, 21(116):1–6, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset collection used for all evaluation experiments."},{"cited_title":"Measuring Time-Series Dataset Similarity using Wasserstein Distance","cited_arxiv_id":"2507.22189","evidence_quote":"Defines the Wasserstein dataset-similarity metric and motivates its connection to out-of-distribution and transfer loss."},{"cited_title":"Match-and-deform: Time series domain adap- tation through optimal transport and temporal alignment","cited_arxiv_id":null,"evidence_quote":"Defines the match-and-deform algorithm that combines optimal transport with DTW alignment."},{"cited_title":"Time-moe: Billion-scale time series foundation models with mixture of experts","cited_arxiv_id":null,"evidence_quote":"Supplies the other forecasting foundation model used in out-of-distribution and transfer-learning evaluation."}],"review_version":1}