{"id":"e6262f3f-6cfe-4adf-a623-3dc29f8ef855","arxiv_id":"2501.11730","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":15,"one_line_summary":"A spectral transformer called SSF forecasts railway axle vibration signals with lower MSE than a time-domain transformer baseline on a laboratory bogie test rig dataset.","lead":"This paper trains transformer models to forecast vibration signals from railway axle test rigs, using signals from healthy and cracked axles. The authors report that their spectral transformer, SSF, predicts signals with lower error than their simpler model, which could support predictive maintenance and reduce costly inspections.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 may compare time-domain MSE for SF with frequency-domain MSE for SSF, so the claimed 3.3x improvement is not yet established; the same MSE values in time- and frequency-domain captions suggest an inconsistent metric definition.","rationale":"I agree with the reader's CONDITIONAL verdict and with the concern that the evaluation is under-specified, but I identify a sharper internal-validity issue than the reader's stated weakest assumption. The reader's weakest assumption is external representativeness of the lab rig and FEM simulations; my concern is that the two performance numbers in Table 3 may not be measuring the same thing at all. Section 2.2 defines SF's loss in the time domain, while Section 2.3 explicitly defines SSF's loss in the frequency domain. Without a stated Parseval/tight-frame relation or explicit common normalization, the 0.5303 vs 1.73285 comparison could reflect domain scaling rather than forecasting quality. The identical MSE values in the time- and frequency-domain captions (Figures 9-12) are a red flag that the metric is either being reused without justification or computed in a way that obscures the domain mismatch. A single recomputation on a common representation would settle this. If the comparison survives that check, the remaining concerns about baselines, error bars, per-fault generalization, and rig representativeness remain appropriate conditions on the claim, so the reader's CONDITIONAL verdict is unchanged.","tokens_in":10043,"tokens_out":9319,"duration_ms":99049,"concrete_test":"Recompute test MSE for SF and SSF on a common representation with identical normalization and the same split: for SSF, apply iSTFT to the predicted spectrogram and compute time-domain MSE against the original signal; for SF, compute the STFT of its time-domain prediction and evaluate frequency-domain MSE using SSF's STFT parameters. If the SSF advantage over SF is not reproduced on both representations, Table 3's central comparison is invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the SSF-vs-SF comparison in Table 3, but the two models are not obviously evaluated on the same quantity. Section 2.2 states that SF operates on the time-domain signal and that its loss is MSE in that domain. Section 2.3 states that SSF operates on the STFT spectrogram and, explicitly, 'we evaluate the loss function in the frequency domain', even though the signal is transformed back with iSTFT. Table 3 then reports a single 'MSE loss' for both models without stating whether the reported values are time-domain MSE, normalized-spectrogram MSE, or something else. Under the default reading, SF's test loss (1.73285) is time-domain MSE while SSF's test loss (0.5303) is frequency-domain MSE; those are different quantities and need not be comparable unless the STFT is a tight Parseval frame with identical scaling and normalization. This is not a minor presentational issue: the paper's headline result is a 3.27x improvement, and if the losses are computed on different scales the improvement can be an artifact of the representation. The suspicion is reinforced by Figures 9-12, where the same MSE values (0.18 and 0.55) are printed in both frequency-domain and time-domain captions without any stated equivalence. No error bars or repeated-seed variation are provided either, so the gap cannot be attributed to genuine forecasting superiority rather than metric mismatch or run-to-run noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two deep autoregressive transformer models for forecasting railway axle vibration signals measured on a bogie test rig: ShaftFormer (SF), a time-domain Informer-based model with a locality-enhancement preprocessing step, and Spectral ShaftFormer (SSF), which operates on STFT spectrograms, splits attention into high- and low-frequency groups (HiLo attention), conditions on finite-element (Abaqus) simulation data, and uses an exponential-variance sampling scheme for the predictive normal distribution. The central empirical claim is in Section 3.3 (Table 3): SSF 'significantly outperforms' SF, with test MSE 0.5303 versus 1.73285, and the predicted spectrograms 'closely match' the true ones (Figures 9-12). The abstract further claims that the approach simulates and predicts vibration signals 'under various conditions and fault scenarios,' improving dataset robustness for detection systems. The appendices document TPE-based hyperparameter optimization, positional-encoding ablations, and the differentiable sampling schemes.","tokens_in":10474,"tokens_out":6138,"duration_ms":57603,"significance":"If the central comparison were properly established, SSF would represent a credible application of spectral attention and frequency-domain generative modeling to railway axle condition monitoring, and the open-source implementations (two GitHub repositories) together with a real bogie-rig dataset would be useful community assets. The exponential-variance observation model and global frequency filtering extend the authors' prior spectral attention work (Moreno-Pino et al. 2023) with HiLo attention and FEM conditioning. However, as reported, the evidence consists of a single table with two models, one MSE value per split, no baselines, no repeated-seed variation, and a metric whose domain is ambiguous; the abstract-level claims about simulation, fault-scenario coverage, and detection are not backed by any experiments in the paper. The reproducible artifacts and the experimental dataset are genuine strengths; the empirical demonstration is the weakness.","major_comments":[{"comment":"The headline result is not currently measurable because the two models appear to be evaluated on different quantities. Section 2.2 states that SF processes the time-domain signal and uses MSE loss there, while Section 2.3 states for SSF that 'we evaluate the loss function in the frequency domain,' even though the signal is transformed back via iSTFT. Table 3 reports a single 'MSE loss' for each model without stating the domain or the STFT normalization. Time-domain MSE and normalized spectrogram MSE are different quantities, comparable only if the STFT/iSTFT pair is Parseval-tight with identical scaling, and no such statement appears. The claimed 3.27x improvement can therefore be an artifact of representation scale. Please report both metrics in both domains for both models, or state explicitly the exact loss definition and STFT normalization used for each number in Table 3.","section":"§2.3, §3.3 (Table 3)"},{"comment":"Figures 9-12 print identical MSE values (0.18 validation, 0.55 test) in both the spectrogram captions and the time-domain captions, with no stated equivalence between the two domains. This is coherent only if the two quantities coincide by construction, which would need to be proved and stated, or if one set of captions is wrong. Please specify which MSE is actually computed and label the domain in each caption; as printed, these figures reinforce rather than resolve the metric ambiguity of Table 3.","section":"§3.1 (Figures 9-12)"},{"comment":"The statement that SSF 'significantly outperforms' SF is unsupported as stated: each cell is a single run with no error bars or repeated-seed results, no statistical test is performed, and no independent baselines (e.g., ARIMA, LSTM, plain Informer, or the spectral attention model of Moreno-Pino et al. 2023) are included. Given the stochastic sampling scheme and the TPE-based hyperparameter search, run-to-run variation and selection effects could plausibly account for the gap. In addition, SF's test loss (1.73285) is roughly four times its validation loss (0.43524), a generalization drop that the paper neither reports nor discusses; this pattern needs an explanation before the comparison can be interpreted.","section":"§3.3 (Table 3)"},{"comment":"The abstract and introduction state that the approach reduces the 'high cost of obtaining experimental signals for all scenarios' by simulating and predicting vibration signals 'under various conditions and fault scenarios,' and that this improves 'dataset robustness for more effective detection systems.' However, no results are broken down by crack level (D0-D3), load (4t/10t), or speed (20/50 km/h), and there are no detection or classification experiments anywhere in the paper. Either report condition-wise forecasting results and detection baselines, or revise the claims to match what is actually demonstrated, namely aggregate forecasting on one held-out split of a single test condition.","section":"§1, §3"},{"comment":"The FEM conditioning of SSF is listed as one of the four design contributions, but the manuscript gives no description of how the Abaqus-generated signals enter the model (whether they condition the observation model, the encoder features, the decoder, or something else) and no ablation showing its effect on the reported MSEs. Without an ablation the reader cannot attribute any part of SSF's performance to this component; please add such an experiment or remove the claim from the list of contributions.","section":"§2.3"}],"minor_comments":[{"comment":"The 'Best Value' for Time Resolution (STFT) is 7, which lies outside the stated search range [8, 16]; correct the table or the range, since out-of-range 'best' values block reproduction of the tuning procedure.","section":"Appendix C, Table C.4"},{"comment":"The 'Damaged Depth (%)' column lists 0.03, 0.06, and 0.08, but 5.7/170 ≈ 3.35%, 10.9/170 ≈ 6.41%, and 15/170 ≈ 8.82%; the column appears to report proportions while being labeled as percentages. Fix the units or the values.","section":"Table 2"},{"comment":"The claim that 'The model performs similarly regardless of the used dataset' is given without any per-dataset numbers; report the three WS1/WS2/WS3 results or delete the sentence.","section":"Appendix C"},{"comment":"The text is inconsistent about the autoregressive target: Section 2.2 says SF generates a distribution 'over the next value of the signal,' while Section 2.3 says SSF generates a distribution 'over the next time window of the spectrogram'; clarify the prediction horizon and how it maps between the two domains.","section":"§2.2, §2.3"},{"comment":"Figures B.16 and B.17 support a comparison with and without positional encoding, but no quantitative result accompanies the claim that the model 'still performs well' without PE; add the corresponding MSEs or soften the claim.","section":"Appendix B"},{"comment":"The paper contains several language and consistency errors that should be corrected, including 'autorregressive' (should be 'autoregressive'), 'Enhancion' (should be 'Enhancement'), and the mismatch between the test MSE 0.5303 in Table 3 and 0.55 in the captions of Figures 10 and 12, which should be reconciled.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is that Table 3 may be comparing time-domain MSE (SF) against frequency-domain MSE (SSF), which would make the headline 3.27x improvement unreviewable as reported; I would require the authors to recompute both models on an identical, fully specified loss and to report errors bars over seeds. The abstract also overclaims relative to the experiments: there are no detection results, no condition-wise breakdown, and no baselines. If the corrected comparison still shows a large gap, this could be a reasonable fit for an applications journal, but the current experimental section is too thin for the claimed significance. I also flag the internal inconsistencies in Table C.4 (best value outside the range) and Table 2 (percent units), which suggest the manuscript needs a careful numerical revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a real dataset and a reasonably assembled architecture, but the central number you'd quote from it—SSF beats SF by 3.3x on test MSE—is not trustworthy as reported. The same MSE values (0.18, 0.55) appear in both frequency-domain and time-domain captions, and Section 2.3 explicitly says SSF evaluates loss in the frequency domain while Section 2.2 says SF is time-domain. Table 3 reports a single 'MSE loss' without saying which quantity it is. Unless the STFT is a normalized tight frame and this is stated, the two losses aren't comparable. This isn't a cosmetic issue; the entire empirical case rests on that comparison.\n\nWhat's genuinely useful: a new experimental dataset from a bogie test rig with three wheelset assemblies, four crack depths, two loads and speeds. The SSF model is a reasonable combination of known pieces—ProbSparse attention, HiLo split, spectral filtering, FEM conditioning—and the authors provide code and hyperparameter details. The qualitative results and STL decomposition suggest the model can capture the main signal structure.\n\nSoft spots beyond the metric: no baselines other than SF, no error bars or repeated-seed runs, so 'significantly outperforms' is unsupported. The abstract promises fault detection and data augmentation, but neither is demonstrated; this is forecasting only. FEM conditioning is never ablated, so you can't tell what it contributes. The leap from a constant-load, constant-speed lab rig to 'diverse operating conditions' in real railways is asserted rather than argued.\n\nWho should read it: researchers in predictive maintenance and transformer-based time-series forecasting who want a worked example on an unusual domain dataset. Treat it as a proof-of-concept, not a benchmark result.\n\nRecommendation: send it to peer review, but require the authors to state exactly which domain each MSE is computed in, reconcile the captions, add at least one standard baseline (LSTM, Informer, or a simple spectral baseline), and report seed variance. The dataset alone is worth refereeing.","headline":"The SSF-vs-SF comparison in Table 3 likely mixes frequency-domain and time-domain MSE, so the paper's headline 3.3x improvement is not established; the dataset and code are real, but the central claim needs clarification and proper baselines.","tokens_in":10959,"tokens_out":2678,"would_cite":false,"duration_ms":27437,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a frequency-domain transformer model, SSF, forecasts railway axle vibration signals with a test mean-squared error of 0.5303, about a third of the 1.73285 achieved by its time-domain counterpart.","keywords":["Maintenance 4.0","Machine Diagnosis","Vibration signal","Time series forecasting","Railway axles","Deep Learning","Transformers","Spectral analysis"],"falsifier":"Run the trained SSF on vibration recorded from an axle in real service, or from the test rig at a speed or load outside the two combinations used in training, and compare its forecasting MSE with the reported test loss; if the error is close to the time-domain SF's 1.73 rather than the reported 0.53, the model's generalization claim is falsified.","tokens_in":9903,"feed_emoji":"🚆","tokens_out":13983,"duration_ms":109404,"temperature":0.7,"pith_summary":"This paper is trying to establish that a transformer-based forecasting model can predict railway axle vibration signals well enough to support condition-based maintenance. The authors test two models: ShaftFormer (SF), which works on raw time series, and Spectral ShaftFormer (SSF), which works on spectrograms computed by a short-time Fourier transform and is conditioned on finite-element simulations of cracked shafts. On experimental data from three wheelset assemblies with crack depths from 0 to 15 mm, SSF achieves a test mean-squared error of 0.5303, compared with 1.73285 for SF, and its predicted spectrograms closely track the measured ones. If this holds beyond the lab rig, it would let railways forecast vibration signals for crack severities and operating conditions they have never measured, impute missing sensor data, and flag anomalies before axle failure.","feed_headline":"Spectral transformer cuts axle-vibration forecast error by two-thirds","feed_subtitle":"Spectrogram forecasts support predictive maintenance, data repair, and fault simulation.","key_machinery":"The load-bearing object is the SSF's spectrogram encoder: the input signal is converted to an STFT spectrogram whose real and imaginary parts are treated as separate channels, and the encoder splits attention heads into a high-frequency group working at full resolution and a low-frequency group working on a 2-D average-pooled version (HiLo attention). A global filtering module selectively mutes frequency bands, and a finite-element conditioning step aligns the model with simulated crack signatures. The output is sampled via a reparameterized normal distribution whose variance is drawn from an exponential distribution, a scheme that keeps generated signals close to the mean. Together these parts let the model produce future windows of the spectrogram rather than raw samples, which is what allows it to preserve the spectral content of the vibration.","core_discovery":"The central claim is that the Spectral ShaftFormer (SSF), a transformer that operates on the short-time Fourier transform of axle vibration and splits its attention into high- and low-frequency branches, forecasts vibration signals substantially more accurately than the time-domain ShaftFormer (SF). Over the three recorded wheelset-assembly datasets, SSF obtains a test loss of 0.5303 while SF obtains 1.73285, and the paper reports that the predicted spectrograms closely match the true ones in both validation and test phases. The model is trained autoregressively in the frequency domain, evaluates its loss on spectrograms, and incorporates an observation model in which the variance of the Gaussian output is itself sampled from an exponential distribution, biasing predictions toward the mean to avoid outliers. The authors also condition SSF on finite-element simulations of cracked-shaft dynamics, so the model is expected to reproduce the harmonic peaks that theory associates with cracks at the rotational speed and its multiples.","pith_inferences":["A natural next test is to run SSF on vibration from an axle in real service, since the paper's experiments stop at the laboratory rig; the model's value for predictive maintenance ultimately depends on that transfer.","The exponential-variance sampling scheme makes the model conservative, so extreme events may be smoothed in forecasts; outlier detection would likely need to be calibrated on the residual distribution instead of on the predicted variance.","The same spectral-attention recipe could apply to other rotating machinery—wheel bearings, gearboxes, drive shafts—whose faults also show up as harmonics of the rotational frequency, so the method is not obviously rail-specific.","A direct test of the model's physical understanding would be to forecast the vibration of a crack depth between the trained values (for instance 8 mm) and check whether the rotational-frequency harmonic scales smoothly with depth, as crack theory predicts."],"forward_implications":["Axle vibration spectrograms can be forecast one window at a time with test MSE near 0.53, so the model can serve as a signal simulator for crack scenarios and operating conditions that were never run on the rig.","Because the model is autoregressive and probabilistic, it can fill gaps in sensor data by predicting the missing windows and preserve the spectral character of the signal.","Deviations between predicted and observed spectrograms can be flagged as outliers, giving the model a role in real-time anomaly detection and early crack warning.","The frequency-domain design carries over to the three wheelset assemblies tested and to both rotation directions, indicating that the architecture's gains do not depend on a single dataset.","The modelling pipeline, from FEM-conditioned training to spectral forecasting, is a step toward a bogie digital twin that can simulate the effect of future damage."],"supporting_citations":[{"why":"Supplies the transformer encoder-decoder and self-attention machinery that both SF and SSF are built on.","marker":"(Vaswani et al., 2017)"},{"why":"Supplies the Informer architecture and ProbSparse attention that SF extends and SSF inherits.","marker":"(Zhou et al., 2021)"},{"why":"Supplies the HiLo attention mechanism that splits SSF's encoder into high- and low-frequency branches.","marker":"(Pan et al., 2022)"},{"why":"Supplies the global frequency filtering and spectral autoregressive approach that SSF adapts for its decoder.","marker":"(Moreno-Pino et al., 2023)"},{"why":"Provides the experimental bogie rig protocol and vibration signals from cracked railway axles that form the datasets.","marker":"(Gómez et al., 2018)"},{"why":"Provides the earlier railway-axle condition-monitoring approach and the rig data on which the forecasting task is posed.","marker":"(Gómez et al., 2020)"},{"why":"Provides the theoretical link between cracks and spectral peaks at rotational speed and its harmonics, used to interpret and condition the FEM simulations.","marker":"(El Arem, 2019)"},{"why":"Supplies the reparameterization trick that makes the Gaussian and exponential sampling steps trainable.","marker":"(Kingma and Welling, 2013)"},{"why":"Supports the design choice of omitting positional encodings, which the authors verify empirically.","marker":"(Haviv et al., 2022)"}],"fun_headline_variants":["Spectral transformer cuts axle-vibration forecast error by 69%","Frequency-split transformer three times better at axle vibration forecasts","Spectral attention on train axles reduces vibration error by two-thirds","Crack signatures predicted by spectral transformer for rail safety","Spectral ShaftFormer forecasts axle vibrations with two-thirds less error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole claim depends on the laboratory bogie rig—three wheelset assemblies at two constant speeds and loads—and the finite-element crack simulations standing in for real axle vibration under the varied operating conditions the paper says its models cover.","fun_headline_variants_meta":{"raw":{"variants":["Spectral transformer cuts axle-vibration forecast error by 69%","Frequency-split transformer three times better at axle vibration forecasts","Spectral attention on train axles reduces vibration error by two-thirds","Crack signatures predicted by spectral transformer for rail safety","Spectral ShaftFormer forecasts axle vibrations with two-thirds less error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3131,"prompt_tokens":908,"completion_tokens":2223,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":524,"tokens_out":2223,"duration_ms":21251,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:55:01.039675+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained SSF on vibration recorded from an axle in real service, or from the test rig at a speed or load outside the two combinations used in training, and compare its forecasting MSE with the reported test loss; if the error is close to the time-domain SF's 1.73 rather than the reported 0.53, the model's generalization claim is falsified.","supporting_citations":[{"cited_title":", author Cai, J","cited_arxiv_id":null,"evidence_quote":"Supplies the HiLo attention mechanism that splits SSF's encoder into high- and low-frequency branches."},{"cited_title":", author Olmos, P.M","cited_arxiv_id":null,"evidence_quote":"Supplies the global frequency filtering and spectral autoregressive approach that SSF adapts for its decoder."},{"cited_title":", year 2019","cited_arxiv_id":null,"evidence_quote":"Provides the theoretical link between cracks and spectral peaks at rotational speed and its harmonics, used to interpret and condition the FEM simulations."}],"review_version":1}