{"id":"b5dbf553-a87e-471e-8c5d-f248506fc90a","arxiv_id":"2412.05475","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LSTM-deep ensemble with STD scaling reports R2 > 0.9 and 51-59 percent AUCE reduction on operational OWC-WEC wave height prediction, but the evaluation lacks baselines, error bars, code, and data.","lead":"Using pressure data from a real wave energy converter in Jeju, South Korea, this paper combines an LSTM network with a deep ensemble and a simple calibration step to forecast wave height a few seconds ahead. It reports accuracy above R2 = 0.9 and says calibration improves the reliability of the uncertainty estimates by more than 50 percent.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >50% calibration improvement is not yet credible: Table 5 lists identical scaling factors for different ensemble sizes (e.g., 0.86652 for DE-2/DE-3 and 0.78296 for DE-4/6–10), and the same baseline configuration gives different metrics in Tables 2 and 4.","rationale":"The reader's weakest assumption focuses on the single-site, single-season data as a representativeness issue. That is a legitimate limitation, but it does not directly refute the in-sample measured claim. The more load-bearing problem is that the novel contribution—over 50% uncertainty-quality improvement via STD scaling—rests on Table 5, whose scaling factors contain exact repeated values across different ensemble sizes. Such repetition is implausible if each s is independently optimized, and it makes the reported AUCE reductions unverifiable. The same baseline configuration also yields different numbers in Tables 2/3 and Table 4, further weakening confidence in the headline metrics. Even if R2 > 0.9 is correct, the paper's distinctive claim is the calibration improvement, and that is exactly where the numerical evidence is most internally inconsistent. A single reproduction check—re-optimizing s per ensemble and recomputing AUCE—would settle whether the concern lands. Until then, the central quantitative claim should be considered unverified rather than conditionally accepted.","tokens_in":19763,"tokens_out":9761,"duration_ms":106188,"concrete_test":"Obtain or reconstruct the data pipeline and code; for each ensemble size M = 2..10, independently re-train or use the saved models, optimize s on the validation split by minimizing the NLL in Eq. (8), then recompute test AUCE before and after calibration. Check whether the optimal s values and AUCE reductions reproduce Table 5, and whether M = 5 reproduces both Table 2 and Table 4. Also report RMSE on raw pressure and on converted meters to resolve the unit ambiguity.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that calibration improves uncertainty quality by over 50% (AUCE reduction) while R2 > 0.9. For this to be true, the AUCE values in Table 5 must come from independently optimized STD-scaling factors for each ensemble size. The reported scaling factors are inconsistent with that protocol: DE-2 and DE-3 share s = 0.86652, and DE-4, DE-6, DE-7, DE-8, DE-9, and DE-10 all share s = 0.78296, to five decimals. If s is optimized per ensemble by minimizing Eq. (8) on the same validation split, exact repeated values to five decimals are not plausible unless the ensembles are degenerate or the values were copied/reused. Additionally, the same baseline configuration (M = 5, window = 300, interval = 70) gives R2 = 0.91242 and AUCE = 0.05123 in Tables 2–3, but R2 = 0.91523 and AUCE = 0.05411 in Table 4. Without seed control or error bars, these numbers are not reproducible from the manuscript. A separate unit check compounds the issue: RMSE = 20.646 with min-max normalized data and MAPE = 0.009 is physically inconsistent if the target is in meters, suggesting the RMSE column may actually be in mbar or cm. These internal inconsistencies place the quantitative strength of the central claim in doubt, independently of the data-representativeness concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LSTM-based deep ensemble (LSTM-DE) model with post-hoc standard-deviation (STD) scaling for real-time wave height prediction using univariate pressure-sensor data from a 500 kW OWC-WEC in Jeju, converted to wave height and min-max normalized. The authors report R2 > 0.9 on a held-out test set and reductions in AUCE of 51–59% after calibration across ensemble sizes of 2 to 10, and they support these results with a parametric study over prediction length, window size, interval size, and ensemble size, as well as qualitative checks on four wave regimes. The paper claims this is the first calibrated deep-ensemble application to real-time wave height prediction and frames the contribution toward digital twin operation of WECs.","tokens_in":20077,"tokens_out":2271,"duration_ms":24182,"significance":"If the quantitative claims were internally consistent, the paper would be a useful applied contribution: it is the first to combine LSTM with deep ensembles and STD scaling for wave-height forecasting on real operational data, and the parametric study could give practitioners practical hyperparameter guidance. The paper also ships a concrete probabilistic architecture (Eq. 3) and a calibration recipe (Eq. 8) that are simple to replicate. However, the quantitative strength of the central claims is currently undercut by unresolved inconsistencies in the reported metrics, the scaling-factor table, and the physical units, so the significance cannot yet be assessed as stated.","major_comments":[{"comment":"The same baseline configuration (M=5, window=300, interval=70) is reported with different metrics in two places: Table 2 gives RMSE=20.64611, MAPE=0.00913, R2=0.91242, AUCE=0.05123, while the 'Number of models' block of Table 4 reports RMSE=20.32808, MAPE=0.00899, R2=0.91523, AUCE=0.05411 for the same hyperparameters. Since no error bars, random-seed control, or repeated-run statistics are provided, these two tables cannot both describe a reproducible single run. This is a load-bearing issue for the abstract's R2 > 0.9 and the >50% AUCE claims, because the exact values of the baseline are not reproducible from the manuscript.","section":"§4.3, Table 4 vs. Table 2"},{"comment":"The STD scaling factors in Table 5 are implausible under the stated protocol of Eq. (8), which minimizes NLL on a separate validation set for each ensemble. DE-2 and DE-3 share s=0.86652 to five decimals, and DE-4, DE-6, DE-7, DE-8, DE-9, DE-10 all share s=0.78296 to five decimals, despite having different AUCE values before and after calibration. Exact duplication at this precision is not credible unless the ensembles are degenerate or the values were copied rather than independently optimized. The paper must either show per-ensemble optimization traces or explain why the same s is expected on separate validation sets.","section":"§4.5, Table 5"},{"comment":"The reported RMSE values are physically inconsistent with the stated target. The authors state that pressure (mbar) was converted to wave height in meters and then min-max normalized, but Table 2 reports RMSE=20.646 and MAPE=0.00913 on the test set. If the target were in meters, RMSE ~20 m is implausible for a 3.5 s forecast of OWC chamber water-column height in Jeju winter; if the RMSE is in normalized units or in mbar, then the claim 'wave height prediction' needs the scale and unit of every metric to be stated explicitly. The paper needs to clarify whether RMSE, MAPE, and R2 are computed on normalized or physical scale, because the abstract's 'notable accuracy' is ambiguous without that.","section":"§2.1, §4.1, Tables 2–3"},{"comment":"The paper makes strong comparative claims (DE outperforms BNN and MC dropout; calibration improves UQ; the proposed 'lightweight' architecture is effective) but provides no baseline comparisons. There is no deterministic LSTM, no BNN, no MC dropout, and no existing wave-height forecaster evaluated on the same data. The only comparison is among ensemble sizes 2–10, which cannot support the statement that DE is 'more practical' than BNN or that the proposed architecture is superior to standard LSTM last-state approaches. The central novelty claim 'first calibrated DE for wave height prediction' requires at least one direct comparison to a non-ensemble probabilistic baseline.","section":"§4.2–§4.5"}],"minor_comments":[{"comment":"There is a typo in the text: 'time-seires' should be 'time-series'.","section":"§2.2"},{"comment":"The narrative states 'AUCE decreases 63.1%' for the baseline model, but Table 5 lists a 54.7% decrease for DE-5, which appears to be the same configuration. Please reconcile the percentage or clarify whether the 63.1% refers to a different index or a different run.","section":"§4.5, Fig. 12 vs. Table 5"},{"comment":"No error bars or standard deviations are reported for any metric, despite the acknowledged stochasticity of neural-network training (different random initializations). Since Table 4 and Table 2 already disagree for the same configuration, the authors should report mean ± std over multiple independent runs for all headline numbers.","section":"§4.2–§4.5"},{"comment":"The scaling factor s is defined as minimizing the NLL on a 'separate validation dataset,' but the paper does not state how the validation split is separated from the 10% test split used for Table 2, nor whether the same scaling factor is used for all prediction indices. Please specify the exact validation split and whether s is re-estimated per index or globally.","section":"§2.5, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the topic is relevant, but the reportability of the headline numbers is not yet adequate. The internal inconsistencies in Tables 2/4 and Table 5 are the main blockers; if the authors can rerun with controlled seeds, report means/stds, and provide a physically interpretable unit scale, the paper could become acceptable. I also note that one of the authors (S. Yang) is a co-author of the closely related calibration reference [61], and the manuscript does not discuss the relationship beyond citing it; a short clarifying sentence would help the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick read of arXiv:2412.05475. The paper is an application-first study: LSTM + deep ensembles + STD scaling for real-time wave height prediction using operational data from a 500 kW OWC-WEC in Jeju. The genuinely new bit is applying calibrated DE to this problem with real sensor data, not simulation, and the parametric study linking window size to gravity-wave periods is a nice practical touch. The LSTM architecture uses all hidden states rather than the last one, which is a reasonable design choice for multi-output forecasting.\n\nBut the quantitative claims currently don't hold up. The headline 'increase uncertainty quality by over 50%' rests on Table 5, and that table has internal problems. Identical scaling factors to five decimals for different ensemble sizes (0.86652 for DE-2 and DE-3; 0.78296 for DE-4 and DE-6 through DE-10) are not plausible if s was independently optimized per ensemble. Either the ensembles are degenerate or the values were copied. Also, the same baseline configuration (M=5, window=300, interval=70) gives R2=0.91242 and AUCE=0.05123 in Tables 2-3, but R2=0.91523 and AUCE=0.05411 in Table 4. Without seed control or error bars, those numbers can't be reproduced from the manuscript.\n\nThe RMSE unit issue compounds this. Test RMSE of 20.65 with MAPE of 0.009 is physically inconsistent if the target is in meters; the authors say pressure was converted from mbar to meters, but the column looks like it's still in mbar or in normalized units. A one-line unit statement would fix it.\n\nOther soft spots: no baselines (no deterministic LSTM, MC dropout, or BNN), no error bars over seeds, data unavailable, and the data covers one winter season at one site, so representativeness is limited. The authors acknowledge the data limitation in the future-work section, so that part is at least honest.\n\nIf the numbers get cleaned up, this could be a useful application paper for the WEC control and digital twin community. Right now I'd send it back for major revision: add baselines, repeat seeds, clarify units, fix or explain Table 5, and share code/data if permission allows. The method itself is sound and the central idea is plausible; the evidence just needs to be made trustworthy. I'd give it a serious referee rather than desk reject.","headline":"Plausible application of calibrated deep ensembles to wave height prediction, but the headline numbers don't yet survive scrutiny.","tokens_in":20662,"tokens_out":2091,"would_cite":false,"duration_ms":19761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep-ensemble LSTM plus a single calibration factor predicts wave height up to 3.5 seconds ahead with R2 above 0.9 on real plant data, and cuts its uncertainty-calibration error by more than half.","keywords":["wave height prediction","deep ensemble","uncertainty quantification","uncertainty calibration","STD scaling","LSTM","oscillating water column","digital twin"],"falsifier":"Run the identical fixed pipeline — same hyperparameters, same ensemble size, same STD-scaling procedure — on a dataset from a different season or a different OWC site and check two numbers: test $R^2$ and post-calibration AUCE. If $R^2$ falls below about 0.9, or the AUCE reduction is far below the reported 51–59 percent without re-learning the scaling factor $s$, the reliability claim is tied to the winter Jeju record rather than to the method. A second, arithmetic check: Section 4.5's text reports a 63.1 percent AUCE decrease for the baseline five-model ensemble while Table 5 records 54.7 percent for the same configuration, so a reproduction should settle which figure holds.","tokens_in":19542,"feed_emoji":"🌊","tokens_out":15807,"duration_ms":134882,"temperature":0.7,"pith_summary":"This paper claims that a deliberately simple predictor — a single-layer LSTM whose outputs are combined through a deep ensemble and then corrected by one scalar calibration step — can forecast the water-column height inside an oscillating-water-column wave energy converter (OWC-WEC) up to 3.5 seconds ahead with $R^2 > 0.9$, while making the model's uncertainty intervals match reality: the area under the calibration error (AUCE) falls by 51–59 percent after calibration. The motivation is operational: a wave energy converter must react to incoming waves in real time, and a deterministic forecast without trustworthy confidence bounds can drive the plant into damaging states such as excessive air inflow through the turbine. Using real operational pressure-sensor data from a 500 kW OWC plant on the coast of Jeju, South Korea, rather than simulation or laboratory data, the paper shows that the raw ensemble is systematically underconfident — every learned scaling factor is below 1 — and that a single multiplicative fix aligns the confidence intervals with observed coverage at negligible extra computation. If the claim holds, the recipe of sequence model, ensemble, and one calibration factor offers a ready template for reliable real-time forecasting in other oscillating ocean-energy systems.","feed_headline":"One calibration step makes wave forecasts trustworthy","feed_subtitle":"Deep-ensemble LSTM tops R2 = 0.9 on real Jeju wave data and halves uncertainty calibration error.","key_machinery":"The load-bearing machinery is the LSTM-DE architecture, a chain of three mechanisms plus a design rule. First, the LSTM layer has 70 nodes in a single hidden layer but, unusually, concatenates the entire sequence of hidden states instead of keeping only the last state, and feeds that full temporal summary to two parallel output heads that predict the mean $\\mu_i(x)$ and variance $\\sigma_i^2(x)$ for $m$ future timesteps at once; this multi-output design sidesteps the error accumulation of autoregressive one-step forecasting. Second, the deep ensemble trains $M$ such networks with identical architecture and different random initializations under the negative log-likelihood loss of Eq. (2) and aggregates them with the Gaussian-mixture formulas of Eq. (3), so the reported variance covers both data noise (aleatoric) and model disagreement (epistemic). Third, STD scaling optimizes the scalar $s$ in Eq. (8) on the held-out validation set to minimize NLL and multiplies the predicted standard deviation by it, a post-hoc step that cannot disturb the point forecast. The design rule that carries much of the experimental design is the domain-knowledge heuristic that the input window should span two to three dominant wave periods and the output horizon about a half period — the parametric study confirms that windows beyond 300 timesteps buy only about 0.02 of $R^2$ at roughly three times the training time.","core_discovery":"The central claim is that accuracy and uncertainty quality can be engineered separately: an LSTM with a single hidden layer captures the temporal dynamics of the wave record, a deep ensemble of identically structured networks trained with different random initializations supplies both a predictive mean $\\hat\\mu(x)$ and predictive variance $\\hat\\sigma^2(x)$ through the aggregation of Eq. (3), and STD scaling — a single factor $s$ optimized on the validation split to minimize the negative log-likelihood — rescales the predicted standard deviation after training. On the test split the baseline five-member model reaches $R^2 = 0.91242$, RMSE = 20.64611, and MAPE = 0.00913, with $R^2$ above 0.99 at the 0.5-second horizon and above 0.85 out to 3.5 seconds. After calibration, the AUCE drops by 51 to 59 percent across ensemble sizes 2 through 10, and because every fitted scaling factor has $s < 1$, the uncalibrated ensemble is shown to be uniformly underconfident. The paper further claims that ocean-engineering domain knowledge sets near-optimal hyperparameters: roughly 300 timesteps (15 seconds) of history — two to three dominant gravity-wave periods — predicts 70 timesteps (3.5 seconds, about half a winter wave period) ahead with the reported accuracy at a fraction of the training cost of larger windows, and the calibrated model keeps ground truth inside its 95% confidence band across regular, amplifying, damping, and calm wave segments of the operational record.","pith_inferences":["Editorial inference: the calibration numbers are demonstrated inside a single winter distribution; the decisive test is distribution shift — summer swell, storms, or a second site — and the fact that every fitted scaling factor is below 1 suggests $s$ would need re-estimation rather than transfer, a point the authors concede when they call for multi-year data in future work.","Editorial inference: the 'two to three dominant periods' input rule is frequency-domain reasoning in disguise, so the same architecture-plus-calibration recipe should port to any oscillatory signal with a known dominant period, such as tides, structural vibration, or breathing loads — the paper does not make this claim.","Editorial inference: the paper validates statistical calibration, not operational benefit; a closed-loop test that feeds the calibrated 95% intervals into the plant's control logic and measures whether turbine trips or air-inflow excursions decrease would settle whether the digital-twin promise is realized.","Editorial inference: with MAPE near 0.9 percent and errors growing with horizon, the residual risk is concentrated in the largest peaks, so pairing the calibrated uncertainty with peak-amplitude alerts would directly serve the turbine-protection use case the paper motivates."],"forward_implications":["Short-horizon forecasts are the usable regime: $R^2$ is above 0.99 at 0.5 seconds ahead and remains above 0.85 out to 3.5 seconds, so control decisions should be pegged to the first one to two seconds of the predicted trajectory.","The ensemble can stay small: accuracy is essentially flat from 2 to 10 members, while uncertainty quality stabilizes at about 5 members, making a five-model ensemble the recommended operating point.","Domain knowledge, not brute-force tuning, sets the hyperparameters: a 15-second input window achieves $R^2 > 0.9$, and window sizes up to 1000 timesteps add only about 0.02 of $R^2$ while multiplying training time by about three.","Because the calibration step is post-hoc, scalar, and uses only the validation set, it adds negligible computation in deployment — the property that makes reliable uncertainty usable inside an online control loop or digital twin.","The calibrated 95% intervals widen at wave peaks, troughs, and direction transitions, which are exactly the instants where a deterministic forecast would be most dangerous, so the uncertainty output points at the risky moments for turbine protection."],"supporting_citations":[{"why":"Establishes the deep-ensemble method: NLL-trained members with random initializations, Gaussian-mixture aggregation of Eq. (3), and the recommendation of at least five models.","marker":"[50]"},{"why":"Supplies the STD-scaling calibration procedure of Eq. (8) and the requirement of a separate validation set for fitting the scaling factor.","marker":"[60]"},{"why":"Provides the calibrated deep-ensemble framework for multi-output regression that this paper extends to temporal multi-output wave prediction.","marker":"[61]"},{"why":"Defines the calibration-curve and AUCE methodology used to score the quality of the uncertainty estimates.","marker":"[57]"},{"why":"The prior variational-Bayesian wave prediction work that supplies uncertainty without calibration, the baseline this paper contrasts with.","marker":"[52]"},{"why":"The LSTM cell the temporal predictor is built on, cited for solving the vanishing-gradient problem in long sequences.","marker":"[66]"},{"why":"Domain source for the rule that two to three dominant wave periods of past data are needed for prediction, which sets the 300-timestep window.","marker":"[17]"},{"why":"Supplies the gravity-wave period band (1–30 seconds, energy concentrated at 5–15 seconds) used to justify window and interval sizes.","marker":"[25]"},{"why":"Motivates the December–February data window by identifying winter as the suitable wave-energy operational season in the Jeju region.","marker":"[74]"}],"fun_headline_variants":["Calibration boosts AI wave forecast uncertainty quality by 50%","Deep ensemble LSTM delivers calibrated wave heights on real ocean data","Wave digital twin: calibration lifts uncertainty quality over 50%","Single calibration step makes wave forecasts 50% more reliable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single winter season of pressure readings from one sensor at the bottom of one OWC chamber, converted to wave height hydrostatically, is a sufficiently representative and stationary signal — if those three months at that one site do not stand for other seasons and locations, the reported $R^2$ and calibration gains do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Calibration boosts AI wave forecast uncertainty quality by 50%","Deep ensemble LSTM delivers calibrated wave heights on real ocean data","Wave digital twin: calibration lifts uncertainty quality over 50%","Single calibration step makes wave forecasts 50% more reliable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3471,"prompt_tokens":1112,"completion_tokens":2359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":2289}},"tokens_in":728,"tokens_out":2359,"duration_ms":17979,"temperature":1.0,"reasoning_tokens":2289,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:41:06.437511+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical fixed pipeline — same hyperparameters, same ensemble size, same STD-scaling procedure — on a dataset from a different season or a different OWC site and check two numbers: test $R^2$ and post-calibration AUCE. If $R^2$ falls below about 0.9, or the AUCE reduction is far below the reported 51–59 percent without re-learning the scaling factor $s$, the reliability claim is tied to the winter Jeju record rather than to the method. A second, arithmetic check: Section 4.5's text reports a 63.1 percent AUCE decrease for the baseline five-model ensemble while Table 5 records 54.7 percent for the same configuration, so a reproduction should settle which figure holds.","supporting_citations":[{"cited_title":"Evaluating and calibrating uncertainty prediction in regression tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the STD-scaling calibration procedure of Eq. (8) and the requirement of a separate validation set for fitting the scaling factor."},{"cited_title":"Towards reliable uncertainty quantification via deep ensemble in multi-output regression task","cited_arxiv_id":null,"evidence_quote":"Provides the calibrated deep-ensemble framework for multi-output regression that this paper extends to temporal multi-output wave prediction."},{"cited_title":"Phase-resolved real-time ocean wave prediction with quantified uncertainty based on variational bayesian machine learning","cited_arxiv_id":null,"evidence_quote":"The prior variational-Bayesian wave prediction work that supplies uncertainty without calibration, the baseline this paper contrasts with."},{"cited_title":"In-ocean validation of a deterministic sea wave prediction (dswp) system leveraging x-band radar to enable optimal control in wave energy conversion systems","cited_arxiv_id":null,"evidence_quote":"Domain source for the rule that two to three dominant wave periods of past data are needed for prediction, which sets the 300-timestep window."},{"cited_title":"Ocean waves and oscillating systems: linear interactions including wave- energy extraction, volume 8","cited_arxiv_id":null,"evidence_quote":"Supplies the gravity-wave period band (1–30 seconds, energy concentrated at 5–15 seconds) used to justify window and interval sizes."},{"cited_title":"Response of wave energy to tidal currents in the western sea of jeju island, korea","cited_arxiv_id":null,"evidence_quote":"Motivates the December–February data window by identifying winter as the suitable wave-energy operational season in the Jeju region."}],"review_version":1}