{"id":"09d82fde-3f11-46b8-acbc-7d25d828c4ae","arxiv_id":"2501.05401","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BRATI, a bidirectional RNN plus attention architecture, reports lower imputation error than state-of-the-art baselines on PhysioNet, water quality, and space weather time series.","lead":"This paper introduces BRATI, a deep learning model that fills missing values in multivariate time-series data. It combines bidirectional recurrent networks with attention mechanisms and reports lower errors than existing methods on three datasets, including a new space weather benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The superiority claim relies on Water Quality baseline rows whose RMSE/MAE pattern (RMSE ≈ 1.3–1.4 with MAE ≈ 0.03–0.06) signals non-convergence or a metric artifact; without a fair comparison there, the central claim is not established.","rationale":"The reader's weakest assumption is that the baselines were trained and hyperparameter-tuned to a fair performance level, and the Water Quality anomaly is the concrete place where that assumption is least secure. I agree with that assessment and with the conditional verdict. The paper has a plausibly novel architecture and a thorough experimental matrix, but the central claim of consistent superiority is not verifiable from the presented point estimates alone: the Water Quality rows are internally implausible, and no variance or significance information is reported anywhere. I would not move to REJECT because the PhysioNet and Space Weather results are more plausible and BRATI is usually better on those metrics; the failure mode is under-evidenced comparison rather than a demonstrated internal contradiction in the method. The right action is to keep the CONDITIONAL recommendation and require the authors to fix the baseline training, report repeated-run statistics, and release code so the comparison can be reproduced.","tokens_in":21461,"tokens_out":7143,"duration_ms":75197,"concrete_test":"Re-run the Water Quality MCAR and sequence-missing configurations for Transformer, SAITS, and BRATI using the public SAITS/Transformer codebases (or the authors' released code) with the paper's hyperparameters, but with 5 independent seeds per configuration and a checked convergence criterion (loss curves and validation MAE). Report mean ± std for MAE and RMSE, and inspect the per-sample error distribution. If Transformer/SAITS RMSE drops below ~0.3, or if BRATI's margin is within one standard deviation of the baselines, the claimed consistent superiority is not supported; if RMSE remains ~1.4, the original baseline rows reflect non-convergence or a metric bug and those table entries must be discarded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that BRATI 'consistently outperformed the baseline models' across all datasets and configurations. That claim depends entirely on Tables 2–7, and Tables 4–5 contain an internal red flag: in the Water Quality MCAR rows, Transformer and SAITS report MAE ≈ 0.056–0.061 but RMSE ≈ 1.26–1.45, while the same architectures in the sequence-missing rows report RMSE ≈ 0.15–0.61. On standardized data, MAE of 0.06 with RMSE of 1.4 cannot occur for a converged imputer unless a small number of predictions are catastrophically wrong; the likely explanations are failed training, a bug in the RMSE computation, or a pathological metric artifact. If those baseline runs did not converge, the Water Quality comparison is not a valid test of SoA, and the paper's blanket conclusion is unsupported. The issue is compounded by the absence of any multi-seed or significance testing: even on the other datasets, margins are often tiny (e.g., Space Weather 0.2 range 3-10: MAE 0.0932 vs 0.0953; RMSE 0.2291 vs 0.2317), so the 'consistent' wording cannot be distinguished from seed noise. This is not a disagreement with the field's consensus; it is an inconsistency internal to the reported numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BRATI, a deep-learning architecture for multivariate time-series imputation that combines a bidirectional recurrent GRU layer with multi-head self-attention in two imputation blocks processing the series in opposite temporal directions. The two block representations are combined with learned attention-derived weights, and training uses a joint objective comprising a masked imputation loss, an observed reconstruction loss, and a consistency loss. The authors evaluate BRATI against median imputation, last-value imputation, M-RNN, BRITS, Transformer, and SAITS on three real-world datasets (PhysioNet-2012, Water Quality, and a newly introduced Space Weather dataset) under three synthetic missingness configurations (random MCAR, fixed-length sequences, random-length sequences) at 10% and 20% missing rates, plus ablation experiments. The central claim is that BRATI consistently outperforms all baseline models across all datasets and configurations.","tokens_in":21801,"tokens_out":4731,"duration_ms":45526,"significance":"If the empirical claims held, the contribution would be a solid but incremental architecture that combines well-known components (GRU, multi-head attention, bidirectional processing, and joint-optimization training) and validates them on a broad experimental grid. The paper's strengths are the clear architectural description, the extensive evaluation across 18 dataset-configuration combinations, and the introduction of a new Space Weather dataset that contains naturally MNAR-missing plasma measurements. The ablations (single block vs. full model, learned combination weights vs. simple averaging) are useful and mostly consistent with the design narrative. However, the central 'consistently outperforms' claim is not currently established: the Water Quality tables contain baseline results with implausible MAE/RMSE ratios, no uncertainty quantification is provided for any result, and MNAR performance is claimed but never actually measured. These issues are fixable within the manuscript's scope, but they require substantive re-analysis.","major_comments":[{"comment":"The Water Quality MCAR rows for Transformer and SAITS report MAE values of 0.056–0.061 with RMSE values of 1.26–1.45, and the 0.2 series-5 row for SAITS reports MAE 0.0439 with RMSE 0.6124. Such MAE/RMSE ratios are not plausible for converged imputers on standardized data and indicate failed training, a metric artifact, or an evaluation bug. The text itself notes that the self-attention models 'struggle with convergence' in the MCAR scenario, so these rows cannot be treated as valid state-of-the-art comparisons. Because these rows are part of the evidence for the blanket conclusion that BRATI consistently outperforms the baselines, the comparison is not currently supported. Please re-run the affected baselines with convergence diagnostics (validation-loss curves, longer patience, multiple restarts), report what happened, and either present converged results or explicitly exclude non-converged runs.","section":"Section 4.5, Tables 4 and 5"},{"comment":"No multi-seed runs, confidence intervals, or statistical significance tests are reported for any experiment. Several reported advantages are very small, for example Space Weather 0.2 range 3-10 MAE 0.0932 vs. 0.0953 and RMSE 0.2291 vs. 0.2317, where the gap is on the order of 1–2%. Without repeated runs or significance testing, the wording 'consistently outperformed' cannot be distinguished from seed noise. Please report means and standard deviations over at least five random seeds, or a paired significance test, for the main comparison tables.","section":"Section 4.5, Tables 2–7"},{"comment":"The paper repeatedly claims that the Space Weather dataset 'highlight[s] MNAR scenarios' and 'captur[es] complex missing data patterns, including Missing Not At Random (MNAR) scenarios,' but the actual evaluation protocol in Section 4.1 artificially removes observed values uniformly or in sequences and uses those removed values as ground truth. The original MNAR-missing values have no ground truth and are never scored. Thus MNAR performance is claimed but not measured. Either rephrase the contribution to state that the dataset contains natural MNAR missingness while the experiments evaluate synthetic MCAR and sequence-missing patterns, or design an evaluation that scores MNAR positions by holding out known values from instrument-saturation periods where ground truth exists.","section":"Sections 4.1 and 6"},{"comment":"The loss weights λMIL, λORL, and λcons are never reported and are not included in the hyperparameter search description, even though they directly determine the objective being optimized. Without these values, the experiments are not reproducible and it is unclear whether the weighting was tuned fairly across models. Please report the chosen weights and their search ranges (or state explicitly that they are fixed constants) for every model.","section":"Section 4.4 and Eq. (24)"}],"minor_comments":[{"comment":"The mask notation is inconsistent: Eq. (18) uses M, Eq. (20) uses \\hat M, and the surrounding text refers to 'the input mask M' even though \\hat M was introduced earlier as the mask of the artificially masked input. Please clarify which mask is used in each equation.","section":"Section 3.3.6, Eqs. (18) and (20)"},{"comment":"The sentence 'The dropout is sampled from (0, 0.1, 0.2, 0.3, 0.4, 0.5).' is duplicated in the hyperparameter search description.","section":"Section 4.4"},{"comment":"The attention weights γw are described as 'averaged across all the heads,' but no averaging operation is specified in the equation. Please define how the averaged weights are computed from the per-head attention matrices.","section":"Section 3.3.5, Eq. (15)"},{"comment":"The term 'Discrepancy' is not defined in the paper; earlier Eq. (6) defines the consistency loss via MAE. Please either use MAE explicitly or define Discrepancy.","section":"Section 3.3.7, Eq. (23)"},{"comment":"The sentence 'Experimental results show that the second and third scenarios are considerably more difficult across all the evaluation metrics than the MCAR one, being the last case the most difficult one' states a result before any results are presented; this sentence should be moved to the discussion or rephrased as a preview.","section":"Section 4.1"},{"comment":"The claim that BRATI 'outperforms all baseline methods in almost every metric in all datasets' is vague; either identify the specific exceptions in Tables 2–7 or change the wording to 'in nearly all configurations.'","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is worth a referee's time, but the headline claim needs work before publication.\n\nThe genuinely new thing is the architecture: two imputation blocks running in opposite temporal directions, each with self-attention, a GRU, a cross-attention step, and a feed-forward network, combined by learned attention-derived weights. That specific stacking doesn't appear in the cited baselines (BRITS, SAITS, STING, BiCMTS). The paper also introduces the Space Weather dataset, which could be a useful benchmark if it gets released. On the plus side, the evaluation is thorough in coverage: three datasets, three missing-data patterns, two missing rates, hyperparameter search, and ablations. The method is described clearly enough to reimplement.\n\nNow the soft spots. The most concrete one is in Tables 4 and 5. On Water Quality MCAR, Transformer and SAITS report MAE around 0.06 but RMSE around 1.3–1.4. On standardized data that combination basically does not happen for a converged imputer; it signals failed training or a metric artifact. The authors even write that the self-attention models 'struggled with convergence' in that scenario, but they still treat those rows as valid baselines in the comparison. That is a real problem for the 'consistently outperforms' claim.\n\nSecond problem: there are no error bars or significance tests anywhere. Some of the winning margins are tiny—e.g., on Space Weather 0.2 range 3-10, BRATI's MAE is 0.0932 vs SAITS 0.0953. Without multi-seed runs you can't call that consistent.\n\nThird, the MNAR framing is overstated. The paper says the Space Weather dataset highlights MNAR, but the actual evaluation artificially masks observed values. True MNAR performance is never measured.\n\nFinally, no code or data is released, which undercuts the new-dataset contribution.\n\nNone of this kills the paper. The architecture is sensible and the ablations support the design choices. But the current experiments don't support the strong conclusion. I'd send this to peer review with a clear request: fix or explain the Water Quality baseline rows, add variance estimates (or at least multiple seeds), and temper the MNAR language. If those baseline runs didn't converge, the claim becomes 'competitive, not clearly better'—which is a different paper.\n\nWho should read it: anyone working on imputation architectures, and anyone who wants a cautionary example of baseline evaluation pitfalls. I wouldn't cite it in its current form, but it's a legitimate submission.","headline":"Promising architecture and broad experiments, but the Water Quality baseline numbers don't add up and the lack of error bars makes 'consistently outperforms' an overclaim.","tokens_in":22322,"tokens_out":3368,"would_cite":false,"duration_ms":30056,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that its bidirectional attention-recurrent architecture, BRATI, outperforms existing imputation models across three real-world datasets and three missing-data patterns.","keywords":["time-series imputation","missing data","bidirectional recurrent networks","self-attention","multivariate time series","Space Weather dataset","missing not at random"],"falsifier":"Re-train Transformer and SAITS on the Water Quality dataset with longer patience, verified loss convergence, and the same hyperparameter budget, then re-run all six masking configurations; if either baseline's RMSE drops to or below BRATI's in any configuration, the paper's claim of consistent superiority fails.","tokens_in":21289,"feed_emoji":"📊","tokens_out":5379,"duration_ms":50738,"temperature":0.7,"pith_summary":"The paper tries to establish that combining bidirectional recurrent networks with attention is a better strategy for multivariate time-series imputation than either family alone. It proposes BRATI, which imputes missing values with two opposing blocks whose outputs are fused by learned attention weights, and it evaluates the model on healthcare, water quality, and space weather data under random missing, fixed-length gap, and variable-length gap scenarios. The payoff if true is that downstream analyses of incomplete sensor, clinical, and solar-wind records can rely on more accurate filled-in values instead of deleting or coarsely estimating missing observations. The paper also contributes a new space-weather benchmark in which missingness is tied to storm intensity, a missing-not-at-random pattern.","feed_headline":"BRATI imputer beats 4 deep baselines on missing time-series data","feed_subtitle":"Two-direction attention plus recurrent layers wins across random, fixed-run, and variable-run missing patterns on three real datasets.","key_machinery":"The load-bearing object is the pair of imputation blocks: each block first reweights the embedded, position-encoded input with self-attention, passes it through a GRU, then runs a second attention layer in which the GRU output queries the first attention output, and finally applies a feed-forward network. A sigmoid gate fuses the forward and backward blocks using the input masks and the averaged attention weights from both directions. The training objective sums a masked imputation loss over artificially concealed values, an observed reconstruction loss, and a consistency loss that penalizes disagreement between the two directions. This machinery is what lets the model exploit both temporal order and cross-feature correlations while choosing how much to trust each direction per time step and feature.","core_discovery":"The paper claims that a model with two imputation blocks, one reading the series forward and one reading it backward, each pairing multi-head self-attention with a GRU plus a learned attention-weighted fusion of the two directions, gives lower imputation error than M-RNN, BRITS, a Transformer encoder, and SAITS on the PhysioNet 2012, Water Quality, and Space Weather datasets under three missing-data mechanisms. It further introduces the Space Weather dataset as a benchmark with missing-not-at-random gaps caused by instrument saturation. On the paper's own terms, the central discovery is that the bidirectional recurrent attention architecture, trained with masked imputation, observed-reconstruction, and consistency losses, is a stronger and more robust imputer than existing recurrent-only and attention-only designs.","pith_inferences":["In the paper's Water Quality tables, Transformer and SAITS report RMSE values above 1.3 while their MAE values are around 0.03 to 0.06, a pattern consistent with a few badly predicted points rather than steady error; if those runs did not converge, the comparison there may be unfair and a corrected benchmark could narrow BRATI's reported advantage.","BRATI's fusion gate and consistency loss could be pulled out and attached to any bidirectional imputer, so the architectural idea may generalize beyond the specific GRU and attention stacking shown here.","A natural next test is whether BRATI's advantage holds under higher missing rates than 20 percent and on irregularly sampled series, since both conditions are common in real deployments."],"forward_implications":["On the three datasets tested, BRATI reports the lowest MAE, RMSE, and MRE in every masking configuration, so its advantage is not limited to random missing values but extends to fixed and variable-length missing runs.","The ablation results show that removing either the second imputation block or the learned fusion weights degrades accuracy most on variable-length gaps, indicating that both components carry the performance on the hardest scenario.","The Space Weather benchmark provides a test bed with missingness caused by instrument saturation, which should make missing-not-at-random imputation more accessible to compare.","Because BRATI uses joint masked-imputation and observed-reconstruction losses plus a consistency loss, it can be trained without requiring data where the missing values are known in advance."],"supporting_citations":[{"why":"This supplies the M-RNN baseline and the bidirectional recurrent approach that BRATI extends.","marker":"[YZvdS18]"},{"why":"This supplies the BRITS baseline and the consistency loss that ties the two directions together.","marker":"[CWL+18]"},{"why":"This supplies the SAITS baseline and the joint-optimization training procedure BRATI adopts.","marker":"[DCL23]"},{"why":"This supplies the Transformer attention layers, positional encoding, and training configuration used by BRATI and the Transformer baseline.","marker":"[VSP+17]"},{"why":"This supplies the PhysioNet 2012 dataset used for the healthcare evaluation.","marker":"[GAG+13]"},{"why":"This supplies the Water Quality dataset and the context for a multi-variable imputation comparison.","marker":"[NC22]"},{"why":"This supplies the ACE space probe data behind the new Space Weather dataset.","marker":"[SFM+98]"}],"fun_headline_variants":["BRATI: two-way attention beats deep baselines for missing data","Bidirectional recurrent attention wins time-series imputation","Missing data? BRATI fills gaps better than state of the art","BRATI: forward-backward attention imputes missing series with edge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison stands on the assumption that the baseline models, especially Transformer and SAITS on the Water Quality data, were trained and tuned well enough to represent fair state-of-the-art performance; if those runs failed to converge, the claimed universal superiority is not established.","fun_headline_variants_meta":{"raw":{"variants":["BRATI: two-way attention beats deep baselines for missing data","Bidirectional recurrent attention wins time-series imputation","Missing data? BRATI fills gaps better than state of the art","BRATI: forward-backward attention imputes missing series with edge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000109,"raw_usage":{"total_tokens":1001,"prompt_tokens":845,"completion_tokens":156,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":81}},"tokens_in":461,"tokens_out":156,"duration_ms":2416,"temperature":1.0,"reasoning_tokens":81,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:33.624218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train Transformer and SAITS on the Water Quality dataset with longer patience, verified loss convergence, and the same hyperparameter budget, then re-run all six masking configurations; if either baseline's RMSE drops to or below BRATI's in any configuration, the paper's claim of consistent superiority fails.","supporting_citations":[],"review_version":1}