{"id":"1b1855ac-c71f-45a8-ae60-ce381f32663c","arxiv_id":"2508.00881","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Pre-trained multivariate time-series imputation models frequently return values that violate known relations between variables, and a diffusion-based score can detect and filter these errors.","lead":"This paper gives a formal way to say when a time-series AI model 'hallucinates', meaning it invents values that break the expected relationships between the measured variables. The authors show that popular open-source models do this often, and that a second AI model can spot and reduce these invented values.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CE metric may only detect OOD-ness, not relational violation: in the benchmark all relation-violating pairs are off-manifold, so the reported detection/mitigation gains may not support the relational-hallucination claim.","rationale":"The reader's CONDITIONAL verdict is appropriate, and my concern sharpens their weakest assumption about CE validity. The central claim has two parts: detection and mitigation. Both depend on CE measuring relational hallucination rather than generic distributional shift. In the proposed relational datasets, the two are perfectly confounded because the data generation process places all training points exactly on the relation manifold; any point with non-zero Er is necessarily OOD. Thus the quantitative evidence in Fig. 8 and Tables 1-2 cannot distinguish between a metric that tracks f and a metric that tracks manifold distance. The toy example in Fig. 1 is suggestive but is not evaluated quantitatively on the real datasets used for the headline numbers. I also note a secondary internal inconsistency in the mitigation evaluation: Section 4.3 defines Delta_Er relative to the mean of the sampled responses, then states it 'should be relative to the response from the model with deactivated dropout,' but Table 2 is reported without clarifying which denominator was used. That issue would need to be resolved too, but the OOD confound is more fundamental because it threatens both the detection and mitigation claims. A concrete OOD-but-relation-satisfying test on rWTH would settle the main concern; the paper's code is promised but not yet available, so such a test is currently not independently reproducible.","tokens_in":16114,"tokens_out":8253,"duration_ms":97318,"concrete_test":"Build a held-out set of OOD but relation-satisfying prompt-response pairs for rWTH: take temperature and humidity values outside the training range and compute VPD via f_vpd(T,H) = 0.6108 * exp(17.27*T/(T+237.3)) * (1-H). Compute CE for (i) in-distribution relation-satisfying pairs, (ii) OOD relation-satisfying pairs, and (iii) relation-violating pairs matched in OOD-ness. If CE for group (ii) is as high as for group (iii), the metric is detecting OOD-ness, not relational error; if CE for group (ii) stays comparable to group (i), the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The CE metric in Eq. (9) is a one-step reconstruction RMSE from a diffusion model conditioned on the full prompt-response pair. The paper claims this metric estimates relational error Er = |f(x)|, but in the five relational benchmark datasets the training distribution is confined to the relation manifold f(x)=0 by construction (Section 4). Therefore every relation-violating prompt-response pair is automatically out-of-distribution, and CE's observed sensitivity to Er in Fig. 8 could simply reflect how far the input is from the training manifold rather than how badly f is violated. The only evidence that CE stays low for OOD-but-relation-satisfying inputs is a qualitative 2D heatmap (Fig. 1 right); no quantitative check on rECL, rWTH, rTraffic, rIllness, or rETT is provided. Since the stated motivation is to trust responses to OOD optimization questions that still satisfy the underlying relation, a CE score that is essentially an OOD detector would not specifically measure relational hallucination, and the headline 59.5%/47.7% numbers would lose their intended interpretation. The paper's own Limitations section (Section 5) acknowledges the approach is 'largely intuition-driven and empirical,' reinforcing that this confound is unresolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes definitions of distributional and relational hallucination for multivariate time-series (MVTS) imputation models, and introduces a diffusion-based Combined Error (CE) metric intended to estimate relational hallucination. The CE metric is computed as the RMSE between a prompt-response pair and a one-step reconstruction of that pair by a RePaint-conditioned diffusion model. The authors introduce five synthetic relational datasets derived from standard MVTS benchmarks, report that pre-trained MVTS foundation models (MOMENT and TIMER) have relational error up to 59.5% of a weak baseline, and report that selecting the lowest-CE sample among N model responses reduces relational error by up to 47.7%.","tokens_in":16329,"tokens_out":7870,"duration_ms":82423,"significance":"If the central identification of CE with relational error holds, the paper makes a useful contribution: it provides operational definitions for a previously undefined phenomenon, a computationally cheap detector based on a single diffusion denoising step, and a simple mitigation strategy. The empirical protocol is a strength: results are reported as means and standard deviations over multiple runs on five datasets, the detector is evaluated against ground-truth relational error, and Appendix A compares several alternative metrics. The main unresolved scientific question is whether CE measures relational violation specifically or merely out-of-distribution status; the paper's own Limitations section concedes that the approach is 'largely intuition-driven and empirical.'","major_comments":[{"comment":"The identification of CE with relational error is confounded with out-of-distribution (OOD) detection. In all five benchmark datasets (rECL, rWTH, rTraffic, rIllness, rETT), the third variable is a deterministic function of the first two, so the training distribution lies on the manifold f(x)=0 by construction and every relation-violating prompt-response pair is out-of-distribution. The only evidence that CE stays low for OOD but relation-satisfying inputs is the qualitative 2D heatmap in Fig. 1 (right); no quantitative check on the five relational datasets is reported. Since the stated motivation (farm-optimization example, §1) is to trust OOD responses that still satisfy the relation, the detection and mitigation results in Tables 2, and the headline 59.5%/47.7% claims, could be driven by generic OOD detection rather than by relational hallucination. Please add a quantitative separation of the two axes, e.g., generate OOD test points that satisfy f and report CE versus Er in that subset, or introduce nondeterministic relations that admit in-distribution violations.","section":"§3, Eq. (9); §4; §5"},{"comment":"The one-step CE computation is underspecified. The text says a single denoising step 'the final time-step going t=0' is used, but Eq. (4) defines µ_θ(x_t,t) as the predicted mean of x_{t-1}, not of x_0. It is unclear whether the reconstructed value is µ_θ(x_T,T), the x_0 estimate (x_T - sqrt(1-arα_T) ε_θ(x_T,T))/sqrt(arα_T), or another quantity, and whether the result is averaged over random draws of x_T. Since MCE is the core of both detection and mitigation, please state the exact estimator and the number of noise realizations used; otherwise Tables 2 and the mitigation numbers are not reproducible.","section":"§3, Eq. (9)"},{"comment":"The mitigation evaluation baseline is inconsistent. The definition says ∆Er = E_r^{(j*)} / ⟨E_r⟩, i.e., relative to the mean of the N sampled responses, but the next sentence says it 'should be relative to the response from the model with deactivated dropout.' Please clarify which baseline produced the numbers in Table 2. If the ensemble-mean baseline was used, the reported reduction confounds the benefit of selecting the lowest-CE sample with the benefit of averaging over samples, and the 47.7% claim needs to be re-evaluated against the intended single-response baseline.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"The text says 'relational hallucination is a subset of distributional hallucination' and later says relational hallucination is 'less restricted' than distributional hallucination; clarify whether the subset relation applies to the set of hallucinating pairs or to the set of relation-satisfying pairs.","section":"§2"},{"comment":"The relational error Er is never explicitly aggregated over the 24 time steps; specify whether Er for a data point is the mean over time of |f(x(τ))| or some other aggregation.","section":"§4"},{"comment":"The overlap coefficient depends on histogram binning; state the number of bins used and whether the histogram is computed per task or pooled across tasks.","section":"§4.2"},{"comment":"The text reports reductions 'up to 55.0%' and 'up to 47.7%' but does not state which dataset and task yield these maxima; add this information so the claims can be located in the table.","section":"Table 2, §4.3"},{"comment":"The CE heatmap in Fig. 1 (right) lacks axis labels and a color bar, which makes the claimed low CE in OOD regions not quantitatively inspectable.","section":"Fig. 1"},{"comment":"There is a typo in 'non-determinstic'; please fix to 'non-deterministic'.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of cs.LG, and I do not see concerns about novelty or citation practices. The main technical risk is the OOD confound; if the authors can show quantitatively that CE remains low for OOD inputs that satisfy the relation on the benchmark datasets, I would be comfortable with acceptance. The promised source code should be released with the revision to make the results reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper gives the time-series foundation-model community a useful formalization of relational hallucination and a set of synthetic benchmarks, but its headline metric (CE) probably measures out-of-distributionness rather than relation violation per se. In the five benchmark datasets, training data is confined to the relation manifold f(x)=0 by construction, so every relation-violating point is simultaneously off-manifold. The observed CE-vs-Er correlation in Fig. 8 could therefore be an OOD artifact. The only direct evidence that CE stays low for OOD-but-relation-satisfying points is a qualitative 2D heatmap (Fig. 1 right); there is no quantitative check on rECL, rWTH, rTraffic, rIllness, or rETT. That matters because the motivating use case is exactly OOD optimization questions that still respect the relation.\n\nWhat's genuinely good: the definitions of distributional and relational hallucination are a needed framing, and the five relational datasets (rECL etc.) are a simple, useful benchmark family. Table 1 convincingly shows MOMENT and TIMER violate the ground-truth relations, at roughly 45–60% of a weak baseline. The quartile detection and CE-based filtering are straightforward, and the overlap coefficients in Table 2 are impressively low. The Limitations section is candid about the lack of theory.\n\nSoft spots besides the OOD confound. The CE computation is underspecified: 'a single denoising step (the final time-step going t = 0)' is not precise enough to reproduce, and the sample count N for mitigation never appears in the method text (only '20 runs' in the table caption). The paper also contradicts itself on the subset relation: Section 2 says relational hallucination is a subset of distributional hallucination, then later says relational is 'less restricted' because it extends to OOD regions. Pick one and reconcile. The 'up to 59.5%' phrasing is the maximum model average, not a typical value; the per-model averages in Table 1 are the more honest summary.\n\nBottom line: the empirical observation—pre-trained FMs relationally hallucinate, and simple filtering on a diffusion reconstruction score reduces relational error in these benchmarks—is probably real. But the paper's central mechanistic claim, that CE tracks relational error rather than mere OOD-ness, is unproven. Two or three added experiments (quantitatively measure CE on OOD prompts that still satisfy f, e.g., extrapolated values on the relation) would settle it. As it stands, I'd send it to referees, but it needs substantial revision before the conclusions are trustworthy. I'd bring it to a reading group and would cite it for the datasets and definitions.","headline":"Useful definitions and a benchmark family for relational hallucination in MVTS imputation, but the CE metric's claimed specificity to relation violation over OOD-ness is not established and needs stronger evidence.","tokens_in":16896,"tokens_out":5747,"would_cite":true,"duration_ms":57478,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that pre-trained multi-variate time-series imputation foundation models relationally hallucinate on average up to 59.5% as much as a weak baseline, and that a diffusion-based Combined Error score detects those…","keywords":["hallucination detection","multi-variate time-series","foundation models","time-series imputation","diffusion models","relational hallucination","confidence estimation","question answering"],"falsifier":"Take the rWTH relational dataset and compute both CE and the known relational error $E_r=|f(x)|$ for every prompt-response pair on the held-out test set across the over-constrained, under-constrained, and forecast tasks; if the rank correlation between CE and $E_r$ is not clearly positive on each task, the detection and filtering claims would fail.","tokens_in":15824,"feed_emoji":"🤖","tokens_out":6618,"duration_ms":64193,"temperature":0.7,"pith_summary":"The paper tries to bring hallucination detection from NLP to multi-variate time-series (MVTS) foundation models, arguing that imputation models can be treated as general question-answering systems whose answers need confidence scores. It defines distributional hallucination as an out-of-distribution prompt-response pair and relational hallucination as a prompt-response pair that breaks a ground-truth relation $f(x)=0$, then focuses on the latter. It proposes a Combined Error (CE) metric computed by one-step diffusion reconstruction, and shows on five derived benchmark datasets with known relations that open-source pre-trained MVTS imputation foundation models relationally hallucinate on average up to 59.5% as much as a weak baseline. If the paper is right, practitioners gain a dataset-specific way to detect when an imputed answer is likely to violate relations among variables and to filter sampled answers to reduce such violations by up to 47.7%.","feed_headline":"Diffusion score catches time-series AI hallucinations","feed_subtitle":"A one-step reconstruction error flags relation-breaking imputations; filtering by it cuts errors by up to 47.7%.","key_machinery":"The load-bearing object is the Combined Error metric, $M_{CE}=\\mathrm{RMSE}_i(\\hat{\\hat{x}}_i, \\hat{x}_i)$, computed by taking a completed prompt-response window and asking a RePaint-conditioned DDPM to denoise it one step: all values are corrupted to the final noise level, reconstructed, and the RMSE between reconstruction and original is averaged over variables. A dataset-specific scale is obtained by computing CE on the training set and thresholding at its quartiles, giving low, medium, and high expected hallucination classes. The five relational benchmark datasets carry the evaluation: each adds a third variable as a fixed function of two real variables (difference, nonlinear vapour-pressure deficit, sum, difference, and product), so the ground-truth relational error $E_r=|f(x)|$ can be computed and the CE estimate can be checked against it.","core_discovery":"On the paper's own terms, the central claim is that relational hallucination in MVTS imputation is measurable without knowing the ground-truth relation. The authors define relational error as $E_r=|f(x)|$ for a known relation $f(x)=0$, call a response relationally hallucinating when this error is high, and demonstrate that a diffusion model trained on a target dataset can estimate this error through a Combined Error (CE) metric: the RMSE between a prompt-response pair and its one-step reconstruction by a RePaint-conditioned DDPM. On five derived relational datasets built from electricity, weather, traffic, illness, and transformer-temperature data, two open-source pre-trained imputation foundation models produced relational errors on average up to 59.5% of the weak-baseline error. Quartile thresholding on CE separates low- and high-hallucination groups with near-zero overlap on four of the five datasets, and selecting the sampled response with the lowest CE reduces relational error by up to 47.7% for the pre-trained foundation models. The paper frames relational hallucination as a subset of distributional hallucination that remains detectable in out-of-distribution regimes, which matters for optimization and what-if questions.","pith_inferences":["The paper leaves implicit that CE is a candidate calibration signal: if it tracks relational error, MVTS question-answering systems could report per-answer 'relationally confident' versus 'relationally doubtful' labels rather than only filtering a batch of samples.","The one-step reconstruction cost makes CE cheap enough for online use, so a natural extension would be to train a small regression head on prompt-response pairs to predict CE directly and avoid running a diffusion pass at deployment.","The definitions suggest a two-axis failure taxonomy for MVTS — a response can be in-distribution but relationally broken, or out-of-distribution but relationally valid — and a detector combining an OOD score with CE could separate these cases rather than collapsing hallucination into one number.","A testable extension is to check whether the CE-to-Er mapping is monotone enough to choose risk-based thresholds instead of quartiles; the paper's toy heatmap hints at this, but the benchmark results only establish separation, not monotonicity."],"forward_implications":["MVTS imputation models can be used more safely when responses are filtered by CE: the paper reports up to 47.7% average reduction in relational error for pre-trained foundation models and 55.0% for the diffusion model itself.","Quartile-based CE thresholds separate low- and high-hallucination classes with near-zero overlap on four of five evaluated datasets, so CE can serve as a practical dataset-specific confidence score at inference.","Because CE only needs a diffusion model trained on the target domain plus one denoising step, it can be attached to any imputation model, including deterministic models made stochastic by activating dropout.","Relational hallucination is less restricted than distributional hallucination: an out-of-distribution prompt can still yield a relationally correct response, which is exactly the regime where optimization and scenario-simulation questions live."],"supporting_citations":[{"why":"Supplies the DDPM forward and reverse diffusion processes from which the CE reconstruction is computed.","marker":"[18]"},{"why":"Supplies the RePaint conditioning method that turns prompts into imputations and enables the one-step final denoising step used for CE.","marker":"[32]"},{"why":"Provides the earlier trajectory-variance hallucination metric that the paper tests and modifies before settling on CE.","marker":"[59]"},{"why":"Provides one of the two open-source pre-trained MVTS imputation foundation models whose relational hallucination is measured.","marker":"[10]"},{"why":"Provides the second open-source pre-trained MVTS imputation foundation model evaluated on the relational datasets.","marker":"[11]"},{"why":"Motivates diffusion models as a strong approach for MVTS imputation, supporting their use as detection and mitigation engine.","marker":"[31]"},{"why":"Source data for the rECL relational benchmark, where the third variable is the difference of the first two.","marker":"[60]"},{"why":"Source data for the rWTH relational benchmark, whose nonlinear vapour-pressure deficit relation gives a realistic known f.","marker":"[61]"},{"why":"Source data for the rETT relational benchmark, where the third variable is the product of two transformer readings.","marker":"[64]"}],"fun_headline_variants":["Diffusion score flags time-series hallucinations","One-step diffusion detects MVTS hallucination","MVTS hallucination cut 47.7% by diffusion filter","Relational hallucination score for time-series AI","Diffusion-based detector tames time-series hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Combined Error score — the RMSE between a prompt-response pair and its one-step diffusion reconstruction — actually tracks the unknown ground-truth relational error $|f(x)|$; the paper's own limitations note that this link is intuition-driven and empirical rather than derived.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion score flags time-series hallucinations","One-step diffusion detects MVTS hallucination","MVTS hallucination cut 47.7% by diffusion filter","Relational hallucination score for time-series AI","Diffusion-based detector tames time-series hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000142,"raw_usage":{"total_tokens":1158,"prompt_tokens":924,"completion_tokens":234,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":162}},"tokens_in":540,"tokens_out":234,"duration_ms":3686,"temperature":1.0,"reasoning_tokens":162,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:48:58.554755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the rWTH relational dataset and compute both CE and the known relational error $E_r=|f(x)|$ for every prompt-response pair on the held-out test set across the over-constrained, under-constrained, and forecast tasks; if the rank correlation between CE and $E_r$ is not clearly positive on each task, the detection and filtering claims would fail.","supporting_citations":[{"cited_title":"Timer: Generative pre-trained transformers are large time series models","cited_arxiv_id":null,"evidence_quote":"Provides the second open-source pre-trained MVTS imputation foundation model evaluated on the relational datasets."},{"cited_title":"MTSCI: A Conditional Diffusion Model for Multivariate Time Series Consistent Imputation","cited_arxiv_id":"2408.05740","evidence_quote":"Motivates diffusion models as a strong approach for MVTS imputation, supporting their use as detection and mitigation engine."},{"cited_title":"Weather data","cited_arxiv_id":null,"evidence_quote":"Source data for the rWTH relational benchmark, whose nonlinear vapour-pressure deficit relation gives a realistic known f."},{"cited_title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","cited_arxiv_id":null,"evidence_quote":"Source data for the rETT relational benchmark, where the third variable is the product of two transformer readings."}],"review_version":1}