{"id":"e63df115-fc2f-4b3f-850c-033d4832183c","arxiv_id":"2506.01361","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A synthetic benchmark suite for time-series causal discovery with known ground truths, combining linear and nonlinear dependencies, trends, seasonality, irregular sampling, missingness, and latent confounders.","lead":"TimeGraph is a new suite of synthetic time-series datasets for testing causal discovery algorithms, with known ground-truth causal graphs and realistic features like trends, seasonality, irregular sampling, missing data, and hidden confounders. It aims to give researchers a fair, reproducible way to compare methods such as PCMCI+ and LPCMCI.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported SHD values in Table 2 are inconsistent with the stated evaluation graph size: with 4 variables and max lag 2, SHD cannot reach 51 under the standard Tigramite 4×4×3 link representation, so the evaluation numbers need a computational check.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: Table 2 contains SHD values that exceed the maximum possible for the paper's stated configuration. I sharpened the calculation: for N=4 and tau_max=2, the Tigramite-style summary graph has 4×4×3 = 48 link slots, so an SHD of 51 cannot arise under that representation. The only ways to make the number coherent are (a) a nonstandard graph representation that the paper does not document, or (b) a bug in the SHD computation. Either way, the empirical demonstration in Section 5.2, which is one of the paper's stated contributions, is not currently reproducible or interpretable. This is not a mere presentational issue: the paper's abstract and evaluation section explicitly use these numbers to claim that algorithm performance varies significantly under realistic temporal conditions. The concern can be settled quickly because the source code is public. I did not find a separate, more fundamental flaw in the benchmark generation itself; the dataset suite may well be valuable, but the stated evaluation evidence cannot be relied upon until the SHD anomaly is resolved or the graph encoding is fully specified.","tokens_in":15657,"tokens_out":6919,"duration_ms":82547,"concrete_test":"Run the provided evaluation code for the B1C (Student's t) row in Table 2 and print the shape and entries of both the true and estimated graph arrays. Count the eligible link slots under the stated configuration (N=4, tau_max=2, so at most 48 slots in a Tigramite summary graph). Recompute SHD directly as the number of mismatching entries and compare with the reported 51. If the recomputed value exceeds the maximum allowed by the actual graph encoding, the metric is confirmed inconsistent; if the encoding is a 12-node full DAG with 66 possible edges, the paper must explicitly say so and reconcile the TPR/FDR definitions with that representation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that TimeGraph reveals significant performance variation across algorithms rests on Table 2. That table reports SHD values up to 51 for datasets described as having 4 variables and a maximum lag of 2. In the standard time-series graph representation used by Tigramite, the graph is an array of shape (N, N, tau_max+1), giving at most 4×4×3 = 48 link slots. Unless the paper explicitly defines a different graph encoding (e.g., a full 3-slice DAG with 12 nodes, which would allow 66 edges), an SHD of 51 is impossible under the implied evaluation protocol. The paper does not specify which encoding was used, how SHD was computed, or how TPR/FDR denominators were derived. Because the anomaly appears in the very table used to demonstrate the benchmark's utility, the correctness of the evaluation metric is load-bearing: if the metric is wrong, the headline conclusions about algorithm performance under nonlinear and confounded settings are unsupported, even if the generated datasets themselves are sound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TimeGraph, a synthetic benchmark suite for time-series causal discovery. It describes ten dataset families (A1, A1C, B1, B1C, B2, B2C, C1, C1C, C2, C2C, D1, D1C, D2, D2C, D3, D3C) that combine linear or nonlinear functional forms, Gaussian/Student-t/Laplace noise, regular or irregular sampling, MCAR/block/combined missingness, and optional latent confounders, each claimed to have a known ground-truth temporal causal graph. The authors release generation scripts and demonstrate the suite by evaluating FGES, PC, PCMCI+, and LPCMCI on a subset of the variants, reporting TPR, FDR, and SHD. The main claims are that TimeGraph provides a more realistic and systematic benchmark than existing synthetic resources and that the experiments reveal large performance variations under nonlinearity and confounding.","tokens_in":15848,"tokens_out":12109,"duration_ms":139368,"significance":"If the dataset generation and evaluation metrics are made precise and reproducible, TimeGraph could be a useful community resource: it combines trends, seasonality, irregular sampling, missingness, and latent confounding in a single controlled framework, and the public code release is a concrete strength. The proof-of-concept evaluation also highlights practically relevant failure modes of standard algorithms on nonlinear and confounded time series. However, the manuscript currently does not specify the generative equations or the exact metric protocol, and the reported SHD values are inconsistent with the stated graph size; the empirical illustration therefore needs to be repaired before the benchmark's utility claim is fully supported.","major_comments":[{"comment":"The reported SHD values are inconsistent with the stated graph size. For a dataset with 4 observed variables and tau_max=2, the standard Tigramite graph representation (used in Section 3, 'Implementation and Evaluation') has at most 4*4*3=48 directed link slots (44 if contemporaneous self-loops are excluded). Any SHD computed as the number of edge additions, deletions, and direction reversals over that representation is bounded by the number of slots, so SHD values of 45 (B1 with Student's t error, PCMCI+), 51 (B1C with Student's t error, PCMCI+), and 36 (B1C with Student's t error, LPCMCI) cannot arise under the stated protocol. If an unrolled 3-slice DAG encoding with 12 nodes was used instead, the effective number of variables and the TPR/FDR denominators change, and the paper does not state this. Because Section 5.2's conclusions about algorithm performance on nonlinear and confounded variants rest on this table, the evaluation must be recomputed and accompanied by an explicit definition of the graph encoding and of SHD.","section":"Table 2 / Section 5.2"},{"comment":"The generative models are described only verbally. The paper does not provide structural equations, coefficient values, base graph edge sets, graph densities, or parameter ranges for any of the variants, nor does it specify how the latent confounder U enters the equations and whether its incident edges are part of the ground-truth graph. As a result, the central claim that each dataset is accompanied by a fully specified causal graph cannot be verified from the manuscript without inspecting the code. A benchmark paper should include, at minimum in an appendix, the exact equations, the baseline graph adjacency matrices, and the parameter settings for each variant.","section":"Section 3 / Table 1"},{"comment":"The empirical evaluation is narrower than the paper's claims. Table 2 contains only ten rows, all with 4 variables, lag 2, and 500 samples, and only the A1, B1, and C1 families (with and without confounders) are tested. None of the irregular-sampling variants (A2, B2, C2), missing-data variants (D1, D2, D3), or mixed-error variants is evaluated, despite the abstract and conclusion stating that the experiments reveal performance variation under 'sampling irregularity' and 'missing data patterns.' The authors should either add experiments for those variants or restrict the empirical claims to the variants actually tested.","section":"Section 5 / Abstract and Conclusion"},{"comment":"The definitions of TPR, FDR, and SHD are ambiguous for the confounded variants. For A1C, B1C, and C1C, it is not stated whether the latent variable U is included in the ground-truth graph or whether the ground truth is a projection onto the observed variables; this changes the denominators of TPR/FDR and the set of edges over which SHD is computed. The paper defines the metrics only in words and does not specify counting rules for contemporaneous edges or for direction reversals. This ambiguity is part of the reason the Table 2 numbers cannot be checked from the manuscript alone.","section":"Section 5, metric definitions"}],"minor_comments":[{"comment":"The comparison between the C1 synthetic series and real stock data lacks axis labels, units, and a source for the real data, and the claimed 'qualitative similarities' are not quantified.","section":"Figure 2 / Section 4"},{"comment":"The Gaussian noise is written as 'N(0, 0.12)' without stating whether 0.12 is a variance or a standard deviation; the Student-t degrees of freedom should also be stated consistently in Table 1 and Section 4.","section":"Section 4 / Table 1"},{"comment":"Section 7 says the evaluation operates at 'moderate scale (e.g., up to 8 variables),' but the only reported experimental results are for 4 variables; Figure 3 shows graphs for 6 and 8 variables but no corresponding results. The text should clarify what was actually run.","section":"Section 7 / Table 2"},{"comment":"The description of the PC baseline does not state how it was adapted to time-series data (e.g., lagged-variable expansion) or which conditional-independence test was used; similarly, the FGES scoring criterion is not specified.","section":"Section 5.1"},{"comment":"The dash notation for missing results is not explained, and some rows have formatting issues (e.g., missing spaces between the dataset label and the first metric column).","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The SHD anomaly in Table 2 is likely an implementation-level issue, but it is load-bearing for the empirical demonstration. I would ask the authors to share the exact evaluation script and to confirm the graph encoding and metric definitions used to produce the reported TPR/FDR/SHD values. The absence of explicit generative equations is also central for a benchmark paper; an appendix with full equations and graph descriptions should be required. Once these points are addressed, the dataset contribution itself could be a solid resource for the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: TimeGraph fills a real gap—a synthetic time-series causal discovery benchmark that combines nonlinearity, trends, seasonality, irregular sampling, structured missingness, and latent confounders with known ground truth. That combination is genuinely new, and the public code makes it reproducible. But the empirical section has a metric problem that undermines the headline numbers, and the abstract overstates what was actually evaluated.\n\nWhat's good: the dataset design is more comprehensive than CausalTime, which relies on deep nets and importance-based graph extraction and doesn't handle latent confounding. The authors provide generation scripts, a standardized protocol, and an honest limitations section. The proof-of-concept framing in Section 5.2 is appropriate, not overclaimed.\n\nThe soft spots, in order of severity:\n\n1. The SHD values in Table 2 are impossible under the stated protocol. With 4 variables and max lag 2, a Tigramite-style graph has at most 4×4×3=48 link slots, and actually 44 if you exclude contemporaneous self-loops. SHD 51 and 45 cannot arise from comparing two graphs on that representation. The paper never states the graph encoding used for SHD. If they are comparing full 3-slice DAGs with 12 nodes, that allows up to 66 edges, but they need to say so explicitly. Without that clarification, the metric computation looks wrong, and the main empirical claim—that performance varies significantly—is not supported by these numbers.\n\n2. The abstract claims evaluation across \"a diverse array of configurations,\" but Table 2 only covers A1, B1, C1 and their confounded versions. The irregular-sampling and missing-data variants, which are core contributions, are not evaluated at all.\n\n3. Most functional forms are described verbally, not with explicit equations. The code is available, so this is a self-containment issue, not a fatal one.\n\nThe stress-test note is correct: the SHD anomaly is load-bearing. If the metric is wrong, the proof-of-concept demonstration fails, even if the generated datasets themselves are sound.\n\nWho this is for: researchers who want a synthetic testbed for time-series causal discovery and are willing to inspect the code and verify the graphs. The benchmark could be useful, but it needs a corrected and clarified evaluation before being cited.\n\nRecommendation: send it to peer review. The resource is valuable and the issues are fixable—clarify the graph encoding, recompute the metrics, and either evaluate the missing/irregular variants or soften the abstract.","headline":"Useful benchmark resource with a genuine gap to fill, but the SHD numbers in the evaluation are internally inconsistent and the abstract oversells the scope.","tokens_in":16355,"tokens_out":4216,"would_cite":false,"duration_ms":46503,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TimeGraph supplies synthetic time-series benchmark datasets with known ground-truth causal graphs, and its evaluations show that standard causal discovery algorithms collapse on nonlinear and confounded variants.","keywords":["causal discovery benchmarks","synthetic time series","temporal causality","ground truth temporal datasets","multivariate time series","non-stationary data","missing data","latent confounders"],"falsifier":"Check the results table against the released ground-truth graphs: a 4-variable graph with lags 0, 1, and 2 has at most 48 directed edges, so any reported edge-error score above that bound, such as 45 or 51, would indicate the metric or graph representation is inconsistent with the stated protocol. If those scores do not reproduce from the released scripts, the claim that standard algorithms fail near-randomly on the nonlinear variants would need to be re-evaluated.","tokens_in":15443,"feed_emoji":"📊","tokens_out":9620,"duration_ms":91791,"temperature":0.7,"pith_summary":"The paper introduces TimeGraph, a family of synthetic time-series datasets built so that the true causal graph is always known. The datasets deliberately include realistic complications that most benchmarks leave out: trends, seasonality, irregular sampling, heavy-tailed or mixed noise, random and block missing data, and unobserved confounders. The central claim is that these complications change how well causal discovery algorithms perform, with standard methods that do well on linear-Gaussian benchmarks dropping to near-random performance on nonlinear or confounded variants. If the claim holds, TimeGraph provides a controlled testbed for locating exactly where algorithms fail and for comparing new methods on equal terms.","feed_headline":"New benchmarks expose time-series causal discovery failures","feed_subtitle":"Each dataset has a known true causal graph, isolating trends, seasonality, hidden causes, and missing data.","key_machinery":"The central object is the dataset variant generator, a time-lagged structural equation model with a known ground-truth causal graph. The generator combines four levers: functional form (linear, polynomial, or trigonometric trend-seasonal), error distribution (Gaussian, Student-t, or Gaussian-Laplace mixture), temporal sampling (regular or exponential waiting-time irregular), and missingness mechanism (none, MCAR, block, or combined), plus an optional latent confounder inserted into the equations. The key design move is paired construction: for each functional family the confounded and unconfounded versions share the same observable equations, so any performance gap between them can be attributed to the hidden variable. PCMCI+ and LPCMCI are not part of the generation machinery; they are evaluation probes that use conditional-independence and momentary-conditional-independence tests to try to recover the graph the generator produced.","core_discovery":"TimeGraph is a benchmark suite of synthetic multivariate time series generated from known structural equation models with lagged dependencies. Each dataset family comes in two versions, one with all variables observed and one with an added latent confounder, so the effect of hidden confounding can be isolated while the ground-truth graph stays fixed. The functional forms progress from linear autoregressions to polynomial and trigonometric trend-seasonal models, with variants for irregular timestamps, mixed error distributions, MCAR and block missingness, and sample sizes from 500 to 5,000. The paper reports evaluations of FGES, PC, PCMCI+, and LPCMCI using true positive rate, false discovery rate, and structural Hamming distance, and finds that performance depends sharply on these temporal conditions: linear Gaussian datasets are recovered well, while purely nonlinear datasets drive true positive rate toward zero and false discovery rate toward one. The suite is released with generation scripts and evaluation protocols so that results can be reproduced.","pith_inferences":["Extension: the paired confounded and unconfounded designs could support a single 'confounding robustness' score, such as the performance drop from the unconfounded to the confounded version of each family, which the paper reports only implicitly.","Extension: if the edge-error scores in the results table are inconsistent with the released ground-truth graphs, the paper's specific algorithm rankings should be treated as provisional; the benchmark artifacts themselves, not the rankings, would be the durable contribution.","Extension: the complete failure of all evaluated methods on the polynomial-Gaussian family suggests that variant may be too difficult to discriminate between algorithms; a graded nonlinearity ladder would be a natural next step for benchmark design."],"forward_implications":["On the released linear-Gaussian variants, all four evaluated algorithms recover near-correct graphs, so these variants can serve as sanity checks for new methods.","On polynomial and trigonometric trend-seasonal variants, the same algorithms drop to near-zero true positive rate and near-one false discovery rate, so results on linear or stationary data cannot be extrapolated to realistic temporal settings.","Comparing each dataset family with its confounded counterpart isolates the effect of latent confounding, letting users test whether a method's robustness claim holds.","The irregular-sampling and missing-data variants provide controlled conditions for studying how preprocessing and imputation choices interact with causal discovery.","Because every dataset ships with a fully specified causal graph, true positive rate, false discovery rate, and structural Hamming distance can be computed uniformly across studies, which is what enables cross-paper comparison."],"supporting_citations":[{"why":"Supplies the benchmark tradition and the cause-effect pairs collection that TimeGraph positions itself against.","marker":"[35]"},{"why":"The closest existing time-series benchmark; TimeGraph is contrasted with its realism approach and its lack of latent confounding.","marker":"[11]"},{"why":"Motivates the need for realistic nonlinear time-series benchmarks and documents limitations of linear synthetic designs.","marker":"[48]"},{"why":"Defines the PC algorithm and the constraint-based causal discovery framework used as a baseline.","marker":"[53]"},{"why":"Defines the greedy equivalence search scoring approach used by the FGES baseline.","marker":"[12]"},{"why":"Defines PCMCI+ and the momentary conditional independence tests used to evaluate TimeGraph datasets.","marker":"[47]"},{"why":"Defines LPCMCI, the baseline algorithm designed to handle latent confounders.","marker":"[20]"},{"why":"Supplies the missing-data mechanisms that TimeGraph's MCAR and block-missingness variants implement.","marker":"[32]"},{"why":"Provides the structural equation model and causal graph framework used to generate ground-truth datasets.","marker":"[37]"}],"fun_headline_variants":["TimeGraph benchmarks expose glaring causal discovery gaps","TimeGraph: synthetic benchmarks reveal causal discovery failures","Causal discovery struggles on realistic time series, new benchmarks show","New benchmark suite highlights causal discovery flaws in time series","TimeGraph exposes how causal discovery fails on temporal data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusions depend on the assumption that the scores used to compare algorithms are computed correctly against the true graphs, including the edge-error counts in the results tables.","fun_headline_variants_meta":{"raw":{"variants":["TimeGraph benchmarks expose glaring causal discovery gaps","TimeGraph: synthetic benchmarks reveal causal discovery failures","Causal discovery struggles on realistic time series, new benchmarks show","New benchmark suite highlights causal discovery flaws in time series","TimeGraph exposes how causal discovery fails on temporal data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000692,"raw_usage":{"total_tokens":3144,"prompt_tokens":971,"completion_tokens":2173,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2108}},"tokens_in":587,"tokens_out":2173,"duration_ms":16310,"temperature":1.0,"reasoning_tokens":2108,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:44:02.166226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the results table against the released ground-truth graphs: a 4-variable graph with lags 0, 1, and 2 has at most 48 directed edges, so any reported edge-error score above that bound, such as 45 or 51, would indicate the metric or graph representation is inconsistent with the stated protocol. If those scores do not reproduce from the released scripts, the claim that standard algorithms fail near-randomly on the nonlinear variants would need to be re-evaluated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark tradition and the cause-effect pairs collection that TimeGraph positions itself against."},{"cited_title":"CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery","cited_arxiv_id":"2310.01753","evidence_quote":"The closest existing time-series benchmark; TimeGraph is contrasted with its realism approach and its lack of latent confounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the need for realistic nonlinear time-series benchmarks and documents limitations of linear synthetic designs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the PC algorithm and the constraint-based causal discovery framework used as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines PCMCI+ and the momentary conditional independence tests used to evaluate TimeGraph datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines LPCMCI, the baseline algorithm designed to handle latent confounders."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the missing-data mechanisms that TimeGraph's MCAR and block-missingness variants implement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the structural equation model and causal graph framework used to generate ground-truth datasets."}],"review_version":1}