{"id":"efe23939-399a-4d56-ab76-8dcb8beced1d","arxiv_id":"2505.00473","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A multi-output variant of the Temporal Fusion Transformer, using block-wise masked attention, predicts outputs of three parametric dynamical systems with reported errors mostly below five percent.","lead":"This paper changes a popular forecasting network so it can predict several output quantities from physics simulations at once, and it adds a viewing window that shows which outputs influence each other. The authors test the modified network on three nonlinear model systems and report small prediction errors.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The general claim relies on the initial output vector y(t1) being a sufficient statistic for the future trajectory, which is only guaranteed in the tested examples when outputs equal the full state or the initial state is fixed; this restriction is neither stated nor tested.","rationale":"The reader identified the same load-bearing assumption: y(t1) is treated as a sufficient statistic. This is the single point on which the abstract's universal claim turns. If the synthetic hidden-state test fails, the method is not a general surrogate for parametric dynamical systems but a surrogate for observable-state or fixed-initial-state systems, and the paper must state that restriction. The reader's CONDITIONAL verdict already captures this, so no change is needed; the condition should be an explicit restriction plus a demonstration on a partially observed system with variable initial conditions, or removal of the universality claim. The block-wise masking and interpretability contribution is not affected by this concern, and the empirical results for the three examples remain plausible as narrow findings.","tokens_in":18628,"tokens_out":7729,"duration_ms":88379,"concrete_test":"Run the same iSTFT pipeline on a minimal linear hidden-state system: dx1/dt = a x1 + b x2 + u(t), dx2/dt = c x1 + d x2, output y = x1. Sample parameters and initial conditions randomly, train exactly as in Sections 4.2-4.3 from (mu, y(t1), future u) to the future y sequence, and include test pairs with identical y(t1) and identical parameters but different hidden states. If held-out errors remain at training level, the sufficiency assumption holds for this family; if errors are large for such pairs, the central claim fails because the input does not identify the trajectory. Alternatively, repeat the FitzHugh-Nagumo experiment with random nonzero initial fields sharing y(t1)=(0,0).","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the formulation of Section 2.1, the true initial condition is the full state x(0,mu) in R^N, while the method feeds only the output vector y(t1,mu) = C(mu) x(t1,mu) as the single observed past value and predicts all future outputs in one pass. This is a legitimate map only if y(t1), together with mu and the future input, determines the trajectory. Lorenz-63 satisfies this because y is the full state (Section 4.1). FitzHugh-Nagumo satisfies it for a different reason: the full initial state is fixed at zero, so the model can memorize the map from (mu, input) to the output sequence and the zero initial output adds no variable information (Section 4.2). For the Ferrocyanide model, the output has 3 components while the discretized state has 2003 DOFs, and no argument is given that y(t1) selects a unique state. For a generic parametric system with hidden state and variable initial conditions, the same y(t1), mu, and future input can arise from different initial states that produce different future outputs; the one-step prediction problem is then underdetermined and no transformer can be accurate in general. The abstract's claim 'regardless of nonlinearity ... and dimensionality of the parameter space' therefore overstates the demonstrated scope, and Section 5's statement that 'only the initial condition is needed' equivocates between the full state x(0) and the observed output y(t1).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes iSTFT, a multi-output extension of the Temporal Fusion Transformer for predicting output time series of parametric dynamical systems with time-varying inputs. The main technical ingredients are a data reshaping that aligns multiple outputs into one spatial-temporal sequence and a block-wise masked interpretable multi-head attention that couples outputs while preserving causality. The authors prove that the L1 loss is equivalent to the q=0.5 quantile loss, then evaluate on Lorenz-63 with random initial conditions, a FitzHugh-Nagumo PDE with two parameters and an external input, and a Ferrocyanide reaction model with two parameters. Reported average errors are low (for example, mean errors of 0.0066 and 0.0014 for the FitzHugh-Nagumo outputs under the MAE loss). The abstract, however, claims accuracy regardless of nonlinearity and parameter-space dimensionality, a claim not supported by the experiments.","tokens_in":18976,"tokens_out":5833,"duration_ms":61973,"significance":"If the central claims held, the method would be a useful non-intrusive surrogate for multi-output parametric systems. The extension of TFT's single averaged attention matrix to a block-wise spatial-temporal attention matrix is a reasonable and potentially interpretable architectural choice. The manuscript is transparent about hyperparameters and includes a formal, if elementary, proof of the loss equivalence. The three test problems cover a chaotic ODE and two nonlinear PDE-derived systems with external inputs, which is a sensible initial test bed. However, because no baselines are run and the identifiability of the prediction map from the initial output is not addressed, the demonstrated significance is substantially narrower than the abstract claims; the contribution is an incremental architecture/data-format extension whose general validity remains unproven.","major_comments":[{"comment":"The one-step prediction setup is only well-posed if (y(t1,μ), μ, future input) uniquely determines all future outputs. Eq. (1) defines the initial condition as x(0,μ) in R^N, but the model receives only y(t1,μ)=C(μ)x(t1,μ). Section 4.1 avoids the issue because y is the full state; Section 4.2 avoids it because the initial state is fixed at zero. Section 4.3 has N=2003 and n_o=3 and provides no observability or unique-state argument. For a generic parametric system with hidden state and variable initial conditions, two different states with the same observed output at t1 can lead to different future outputs, so no transformer can be accurate in general. The abstract's claim 'regardless of ... dimensionality of the parameter space' and Section 5's statement that 'only the initial condition is needed' therefore overstate the demonstrated scope. Please either prove or assume an identifiability condition, test on a system with unobserved state and varying initial state, or restrict the claims.","section":"§2.1, Eq. (1); §4.3; §5"},{"comment":"The numerical evaluation contains no baselines. The introduction motivates the work by limitations of LSTM and single-output TFT, and the contribution is an architectural change; yet the paper reports only iSTFT's own errors. Without comparing to retrained single-output TFT models, an LSTM baseline, or a standard multivariate forecaster, the claimed advantage in accuracy and interpretability is not established. Please add baseline comparisons and an ablation that removes the block-wise mask.","section":"§4.1–§4.3"},{"comment":"The claim that accuracy holds 'regardless of ... the dimensionality of the parameter space' is not supported by any experiment with p>2; all three examples have p=2, and Lorenz-63 has random initial-condition parameters but no physical parameters or external input. Please add a higher-dimensional parameter experiment or replace the claim with a qualified statement about the tested regime.","section":"Abstract; §4"},{"comment":"The interpretability analysis is qualitative: it relies on visual inspection of a few attention matrices and variable-importance tables, with no quantitative metric, statistical test, or comparison to an alternative attention mechanism. Since interpretability is a stated main contribution, the current evidence is anecdotal and should be either strengthened or presented as preliminary.","section":"§4.4"}],"minor_comments":[{"comment":"The manuscript uses two different titles, and the acronym 'SE-TFT' appears in the abstract while the body uses 'iSTFT'; please unify the naming.","section":"Title / Abstract"},{"comment":"The error measure switches discontinuously between absolute error and relative error exactly at mean |y|=1, which can make averaged errors across different outputs or examples incomparable.","section":"Eq. (12)"},{"comment":"The text says the outputs at the 'following 500 time instances' are predicted, but only 499 future time instances follow t1; the wording is internally inconsistent.","section":"§4.2"},{"comment":"The notation mixes n_μ, n_p, and n_µ in the data table; please standardize the symbol for the number of parameter samples.","section":"Eq. (2)"},{"comment":"No code or data availability statement is provided; for reproducibility, please include one.","section":"General"},{"comment":"The sentence 'so that if can be used' appears to be a typo for 'so that it can be used'.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The broad claims and the absence of baselines are the main blockers. The core idea is modest but could be publishable after revision and honest recalibration of the scope. The authors should also be asked to position their work against existing multivariate TFT implementations that support multiple targets; the paper argues that the mask and interpretability are the distinction, which is acceptable but should be benchmarked rather than asserted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the block-wise masked attention and the output reshaping are a genuine, simple extension of TFT to multi-output prediction, and the interpretability argument is coherent. The empirical work, though, is much narrower than the abstract claims, and the paper needs a real comparative study before I'd trust \"accurate regardless of nonlinearity and dimensionality.\"\n\nWhat's new: treating the no outputs as an extra dimension and masking the attention in blocks so each block captures output-output interactions at a time step. That is a clean idea and a natural generalization of TFT's interpretable attention. I also credit the authors for reporting per-test-case errors and for checking both L1/MAE and MSE training. The claimed L1 = 0.5 quantile equivalence is textbook—correct, but not a theorem worth restating as a contribution. The citation pattern looks fine; they engage with TFT, the pytorch-forecasting multi-target implementation, and the MOR literature.\n\nWhere it's soft: no baselines at all. The numbers are plausible but unanchored; an LSTM baseline and a plain TFT per output are the obvious minimum. Parameter dimension is two in every example, and the test sets are tiny (6 FHN, 10 Ferro). The abstract's \"regardless of ... dimensionality\" is unsupported and should go. More substantively, the one-step setup assumes the observed output at t1, plus parameters and future input, determines the future output sequence. That holds for Lorenz because output equals full state, and for FitzHugh-Nagumo because the initial state is fixed. It is not argued for Ferrocyanide, where 3 outputs cannot pin down a 2003-dimensional state. The conclusion's \"only the initial condition is needed\" slides between the full state x(0) and the observed output y(t1); that equivocation should be fixed. No code or data is released, which makes the lack of baselines worse.\n\nWho it's for: people building transformer surrogates for parametric ODE/PDE systems and wanting an interpretable multi-output variant. It is a legitimate entry, not a breakthrough. I'd send it to referees—the idea is clean and the exposition is honest enough to warrant a careful look—but I'd condition acceptance on adding baselines, stating the initial-state-sufficiency assumption explicitly, and scaling at least one example beyond p=2.","headline":"A clean, small extension of TFT to multi-output surrogates with a nice attention-mask idea, but the evidence base is too narrow and the abstract overclaims.","tokens_in":19493,"tokens_out":2840,"would_cite":false,"duration_ms":30927,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a transformer with block-wise masked attention can predict all future outputs of a parametric dynamical system in a single pass from the initial output, parameters, and known future input signal.","keywords":["temporal fusion transformer","multi-output forecasting","parametric dynamical systems","time-varying inputs","block-wise masked attention","interpretability","model order reduction","scientific machine learning"],"falsifier":"Take a two-state oscillator with state $(x, v)$ and output $y = x$; generate two trajectories with the same parameters and the same initial output $y(t_1) = 0$ but different initial velocities, train iSTFT with only $y(t_1)$ as the observed past, and check whether it predicts both future output sequences. If the predictions diverge from one of the two trajectories, the initial-output sufficiency assumption is violated and the method's scope is limited to fully observed or fixed-initial-condition systems.","tokens_in":18428,"feed_emoji":"⚙️","tokens_out":10419,"duration_ms":94109,"temperature":0.7,"pith_summary":"The paper claims that a single transformer model, called iSTFT (SE-TFT in the abstract), can predict the full future sequence of multiple outputs of a parametric dynamical system in one forward pass, using only the output at the initial time, the parameter values, and the known future input signal. This would matter because repeated simulation of nonlinear systems with many parameters and external inputs is expensive, and existing surrogate methods typically predict one quantity of interest at a time or rely on lengthy observed histories. The model flattens the multiple outputs into the time dimension and uses a block-wise causal mask for multi-head attention, so the same attention matrix records both temporal correlations and interactions between outputs. The paper demonstrates accurate long-horizon predictions on the chaotic Lorenz-63 system, the FitzHugh-Nagumo neuron model, and a fully coupled Ferrocyanide electrochemistry model, and proves that the original TFT quantile loss at $q=0.5$ is equivalent to the MAE loss used for point prediction.","feed_headline":"A single transformer predicts every system output in one pass","feed_subtitle":"Trained once on black-box dynamics, it forecasts full trajectories from initial outputs and future inputs alone.","key_machinery":"The load-bearing object is the block-wise masked interpretable multi-head attention weight matrix $\\tilde{A} \\in \\mathbb{R}^{(n_o n_t)\\times(n_o n_t)}$, obtained after the data reshaping of Eq. (3) merges the output dimension into the temporal dimension. The causal mask is applied block-by-block instead of entry-by-entry: the block for time $t_i$ and $t_j$ with $i<j$ is masked, while every pair inside an unmasked block remains visible, so the model learns output-to-output interactions at each time step. The matrix therefore carries the temporal correlations of the original TFT and adds the spatial correlations between outputs, which is the mechanism that makes multi-output prediction interpretable in a single pass. The proof of Theorem 3.1 supplies the loss foundation: the quantile loss of TFT at $q=0.5$ equals half the MAE loss, so the MAE point forecasts used here are exactly the original TFT objective up to scaling.","core_discovery":"The central discovery is that reshaping the training table so that the $n_o$ output components at each time step occupy a contiguous block of length $n_o$ turns the temporal fusion transformer into a multi-output predictor, and that masking the attention matrix block-wise preserves and extends interpretability. In the block-wise masked attention matrix $\\tilde{A}$, every entry $\\tilde{a}_{k,l}$ inside the block for time $t_i$ encodes the correlation between output $k$ at $t_i$ and output $l$ at $t_j$, while blocks for future times are masked out as usual. With this single matrix, the model captures both temporal dependence within each output and spatial dependence across outputs, and the variable-selection layer still reports which parameters, past inputs, and past outputs drive the prediction. The paper reports that the overwhelming majority of the chaotic Lorenz-63 test cases, up to about 97 percent, were predicted with error below 5 percent, and that the mean prediction errors for the FitzHugh-Nagumo and Ferrocyanide outputs are mostly in the range of roughly 0.1 to 3 percent when the MAE loss is used, with the MAE loss consistently matching or beating the MSE loss. The paper takes these results as evidence that the approach predicts accurately regardless of system nonlinearity and the dimensionality of the parameter space.","pith_inferences":["If iSTFT is applied to systems whose output is only a partial observation of the state, the initial output vector is generally not a sufficient statistic; the natural repair is to feed a short past window of outputs as observed inputs, and this window length would become a new hyperparameter.","The learned block-wise attention weights could be read as a data-driven coupling matrix for the output variables, offering a justification for decoupling some outputs in reduced-order modeling or sensor placement decisions.","A direct testable extension would compare iSTFT against an LSTM or neural-operator baseline on a system with hidden state but identical initial outputs, which would isolate whether the one-step multi-output property survives partial observability.","Because the discrepancy between attention weights for different outputs was visible in the Ferrocyanide example, where the current density appeared decoupled from the concentrations, similar inspection could be used to detect output groups that evolve almost independently, guiding divide-and-conquer surrogates."],"forward_implications":["One trained iSTFT model replaces a separate TFT model for each quantity of interest, so systems with many outputs no longer require retraining per output.","At test time, only the initial output vector, the parameters, and the known future input are needed; the whole future output sequence is produced in one forward pass, with the three examples taking about 4 to 6.4 seconds for all test cases combined.","The block-wise attention matrix gives a readout of which outputs are coupled and which past values matter most, information that the original single-output TFT cannot show.","The equivalence of the $q=0.5$ quantile loss and the MAE loss means the interpretable multi-horizon forecasting framework can be used directly for accurate point predictions, not only for quantile forecasts."],"supporting_citations":[{"why":"Supplies the temporal fusion transformer architecture that iSTFT extends from single-output to multi-output prediction.","marker":"[32]"},{"why":"Provides the self-attention and multi-head attention formulation, including scaled dot-product attention and the causal masking idea.","marker":"[45]"},{"why":"Shows the existing multi-target TFT implementation whose attention matrix offers no spatial interpretation, the contrast motivating the new block-wise mask.","marker":"[4]"},{"why":"Provides the LSTM-based output prediction approach for parametric systems that iSTFT improves on for long-horizon multi-output prediction.","marker":"[19]"},{"why":"Supplies the FitzHugh-Nagumo benchmark system used as the second numerical example.","marker":"[44]"},{"why":"Supplies the Ferrocyanide oxidation reaction model used as the third numerical example.","marker":"[46]"},{"why":"Provides the biological neuron model underlying the FitzHugh-Nagumo test problem.","marker":"[20]"}],"fun_headline_variants":["One transformer, many outputs: interpretable multi-output prediction","SE-TFT: attention reveals spatial and temporal output links","Transformer predicts all outputs at once, even chaotic ones","97% of chaotic cases within 5% error: transformer does it","Block-wise attention yields interpretable multi-output forecasting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the output vector at the initial time, together with the parameters and the known future input, is a sufficient statistic for the whole future output sequence; this holds in the paper's examples only because Lorenz-63's outputs are the full state and the other two models start from fixed zero initial conditions.","fun_headline_variants_meta":{"raw":{"variants":["One transformer, many outputs: interpretable multi-output prediction","SE-TFT: attention reveals spatial and temporal output links","Transformer predicts all outputs at once, even chaotic ones","97% of chaotic cases within 5% error: transformer does it","Block-wise attention yields interpretable multi-output forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2376,"prompt_tokens":987,"completion_tokens":1389,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1308}},"tokens_in":603,"tokens_out":1389,"duration_ms":11289,"temperature":1.0,"reasoning_tokens":1308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:41:31.506669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a two-state oscillator with state $(x, v)$ and output $y = x$; generate two trajectories with the same parameters and the same initial output $y(t_1) = 0$ but different initial velocities, train iSTFT with only $y(t_1)$ as the observed past, and check whether it predicts both future output sequences. If the predictions diverge from one of the two trajectories, the initial-output sufficiency assumption is violated and the method's scope is limited to fully observed or fixed-initial-condition systems.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the temporal fusion transformer architecture that iSTFT extends from single-output to multi-output prediction."},{"cited_title":"Vaswani, N","cited_arxiv_id":null,"evidence_quote":"Provides the self-attention and multi-head attention formulation, including scaled dot-product attention and the causal masking idea."},{"cited_title":"Beitner and F","cited_arxiv_id":null,"evidence_quote":"Shows the existing multi-target TFT implementation whose attention matrix offers no spatial interpretation, the contrast motivating the new block-wise mask."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LSTM-based output prediction approach for parametric systems that iSTFT improves on for long-horizon multi-output prediction."},{"cited_title":"Fitzhugh-nagumo system","cited_arxiv_id":null,"evidence_quote":"Supplies the FitzHugh-Nagumo benchmark system used as the second numerical example."},{"cited_title":"Vidakovi´ c-Koch, V","cited_arxiv_id":null,"evidence_quote":"Supplies the Ferrocyanide oxidation reaction model used as the third numerical example."},{"cited_title":"FitzHugh","cited_arxiv_id":null,"evidence_quote":"Provides the biological neuron model underlying the FitzHugh-Nagumo test problem."}],"review_version":1}