REVIEW 4 major objections 5 minor 25 references
EPBench: A Benchmark for Short-term Earthquake Prediction with Neural Networks
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read EPBench presents 924,472 earthquakes as the first global benchmark for short-term earthquake prediction.
desk verdict Useful dataset and metric ideas, but the neural output decoding is undefined, so Table 2 cannot be reproduced; worth sending to review to fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the matching rule: for each real earthquake above magnitude 4.5, a window of 2 days and 150 kilometers is opened, and any predicted event inside counts as a match. Matching Rate is the fraction of real events matched, False Alarm Rate is the fraction of predictions with no real event in the window, and ST-MSE is the normalized squared spatio-temporal error of the nearest match, while delta-mag is the magnitude error of that match. This window rule converts continuous prediction outputs into discrete, human-meaningful scores, and it is what lets the benchmark compare a statistical point-process model with CNN, LSTM, Transformer, and diffusion forecasters on equal terms. The region partition by tectonic boundaries and the moment-tensor angular encoding are secondary mechanisms that improve scores in the reported experiments.
What would settle it
Take the benchmark's test windows and artificially delete a random fraction, say 10 to 30 percent, of real events, then recompute matching rate and false alarm rate for ETAS and the neural baselines; if the ranking between families reverses, the reported conclusions are artifacts of catalog completeness.
Extended reading notes
Core claim
The central claim is that EPBench is the first global regional-scale benchmark for short-term earthquake prediction, and that its combination of data partitions, window-based metrics, and baselines makes it possible to evaluate prediction models in a way that existing earthquake datasets do not. In the authors' own experiments, a statistical baseline called ETAS reaches very high matching rates, often above 90 percent, but false alarm rates at or above 90 percent, while neural-network baselines achieve lower matching rates with substantially lower false alarm rates. Adding moment-tensor information to the multimodal setting raises matching rates on several regions, and training on tectonic-region partitions beats training on the whole globe. The paper claims these results show that the benchmark is sensitive enough to expose the real trade-off between coverage and false alarms, rather than declaring any current model ready for operational use.
Load-bearing premise
Everything rests on the raw catalogs being complete enough: if many real earthquakes are missing from a test window, the benchmark's matching rates are underestimated and false alarm rates overestimated, which could change which model looks best.
Editorial extensions
If this is right
- Any future short-term prediction method can be scored on the same data splits and metrics, making published numbers directly comparable.
- To be practically useful, a method must beat ETAS's coverage while keeping false alarms far below 90 percent; EPBench makes that trade-off explicit.
- The reported gains from regional tectonic partitions and from moment-tensor features suggest that physical priors are a promising direction for neural predictors.
- The separate multimodal benchmark for magnitude-6-and-above events creates a harder, more decision-relevant evaluation for high-impact earthquakes.
- The ETAS results indicate that coverage-oriented forecasting alone is not a usable solution, and the benchmark's false-alarm metric captures that.
Reading between the lines
- The authors leave implicit that the reported rankings could shift if the catalogs were corrected for incompleteness: adding missing real events would raise matching rates and lower false alarm rates, and the change would not affect all methods equally.
- A testable extension is to report scores at multiple window sizes; the 2-day and 150-kilometer rule is a single choice, and model rankings may depend on it.
- The benchmark's structure points toward a future version that fuses the 924,000-event catalog with the multimodal records through waveform features, since only the moment-tensor channel is currently used.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents EPBench, proposed as the first global regional-scale short-term earthquake prediction benchmark. It compiles 924,472 catalog records and 2,959 multimodal records from SAGE/USGS, defines regional and temporal partitions, introduces evaluation metrics (MR, FAR, ST-MSE, delta-mag), and reports baseline results for ETAS and four neural network families (CNN, LSTM, Transformer, diffusion). The paper also reports a moment-tensor control experiment and provides code at a public repository. The central claim is that EPBench offers a standardized testbed that the field currently lacks.
Significance. If the evaluation protocol is made fully explicit and reproducible, EPBench would be a useful community resource: the dataset is large, the inclusion of ETAS alongside neural baselines is appropriate, and the MR/FAR/ST-MSE metrics target operational concerns rather than generic time-series losses. The public code release and the honest acknowledgment of limitations in Section 6 are strengths. The main reservations are reproducibility and statistical robustness: the output-to-event decoding for neural models is not documented, a key test window is changed without redefining the metric, and no uncertainty is reported. These issues do not invalidate the benchmark concept, but they currently prevent the reported numbers from serving as a reliable standardized comparison.
major comments (4)
- [F.1, §4.3] The mapping from neural-network output to the discrete predicted event set used by MR and FAR is never defined. F.1 states that all models take 196 time steps of four-dimensional input (magnitude, latitude, longitude, delta_time) and output a four-dimensional tensor with matching channels, while §4.3 defines MR and FAR for a set of predicted events with magnitudes above 4.5 and a spatio-temporal matching window. The paper does not say how many predicted events are emitted, which time steps are used to enforce the three-month horizon, how the magnitude threshold is applied to continuous outputs, or how a fixed-size output is reconciled with a variable number of real events. It is also not stated how irregular catalog times are converted into the 196 fixed-length sequences. Without this decoding step, Table 2 cannot be reproduced from the manuscript, and it is unclear whether ETAS and the neural baselines are evaluated under exactly the same protocol. This is load-bearing because the central contribution is a standardized benchmark; the released code may resolve the ambiguity, but the paper should specify the protocol in full.
- [F.2, Table 3] The distance window used for evaluation changes from 150 km in §4.3 to 2000 km in the moment-tensor experiment described in F.2. Table 3 reports large MR increases when moment-tensor information is added, but a 2000 km window is more than thirteen times larger than the 150 km window used in Table 2, so the comparison is not controlled. The improved MR in Table 3 could be substantially or entirely an artifact of the looser matching threshold. The moment-tensor experiment should either use the same Dw as the main benchmark or report results under both windows so that the claimed effectiveness of the moment-tensor preprocessing can be assessed.
- [§4.2, Table 2] Each region is evaluated on only two non-overlapping three-month test windows (January–March 1995 and January–March 2020), and no error bars, confidence intervals, or bootstrap estimates are provided for any metric. Several entries in Table 2 are based on very small numbers of target events, such as the AfricaAsia rows with MR values of 0%, 6.67%, 20%, and 33.33%. In this setting, point differences of a few percentage points between models cannot be distinguished from sampling noise. A benchmark that is meant to rank prediction methods should report uncertainty quantification or, at minimum, the number of real events in each test window.
- [§4.1, §4.3] The benchmark relies on SAGE/USGS catalog completeness but applies no incompleteness correction, despite citing Mizrahi et al. on data incompleteness. In the test windows, missing or mislocated real events will inflate FAR (because predicted events that would have matched a missing event are counted as false alarms) and can bias MR in either direction. Since completeness varies by region and time, the cross-region and cross-window comparisons in Table 2 are vulnerable. The paper should provide completeness estimates for the test periods or a sensitivity analysis showing that the reported rankings are stable under plausible incompleteness levels.
minor comments (5)
- [F.1, D] There are typos: 'earthuquake' in F.1 and 'Oour' at the start of Section D.
- [Table 2] Table 2 headers read 'EU-CHN(1975∼1995)' and similar, but §4.2 says the first training period is 1970–1995; the notation should be consistent.
- [§1, §4.3] The introduction describes 'false alarms (False Negative) and the omissive reports (False Positive)', which reverses the usual terminology and conflicts with Eq. (3), where FAR counts false alarms; please align the terminology.
- [Abstract, §4.1] The abstract states the time span is 1970 to 2021, while §4.1 says the basic catalog covers 1970 to 2020 and the multimodal data covers 1996 to 2021; clarify which span applies to which data category.
- [Figure 3] The caption refers to 'yellow' and 'blue' models, but the figure may not preserve these colors in print; please add explicit labels or patterns.
Circularity Check
No significant circularity: EPBench evaluates all baselines out-of-sample on future earthquake windows, and no reported metric is forced by construction from the training data.
full rationale
EPBench is a benchmark paper rather than a derivation, and its reported numbers come from genuine out-of-sample evaluation: models are trained on data through 1995, 2020, or 2021 and then scored on subsequent three-month or one-year windows against external SAGE/USGS catalogs. The MR, FAR, ST-MSE, and delta-mag metrics are defined in Section 4.3 independently of any model output, and no baseline parameter is fitted to the test windows; ETAS is fitted to training history and then evaluated prospectively. The paper contains no load-bearing self-citations by the present authors, and no uniqueness theorem or prior-work ansatz is invoked to make a design choice forced. The only substantive concern is that Appendix F.1 does not specify how the 4D output tensor of the neural models is decoded into the discrete predicted-event set required by Equations (2)-(3), which is a reproducibility and standardization gap rather than a circular reduction; it does not make any reported number equivalent to its input by construction. Therefore, no circular step can be exhibited under the required standard.
Assumptions & free parameters
free parameters (6)
- spatio-temporal window (Tw, Dw) =
Tw=2 days, Dw=150 km; 2000 km for M6 experiment
- weighted MSE loss weights =
10x for M>=5, 3x for lat/lon
- sequence length =
196 time steps
- magnitude thresholds =
data >=2.5, evaluation >=4.5, multimodal >=6
- test window lengths =
3 months (two windows); 1 year for M6 experiment
- regional division =
6 hand-defined tectonic regions (EU-CHN, JPN-IDN, AU South, USA-CL, Atlantic, AfricaAsia)
assumptions (3)
- domain assumption The SAGE/USGS earthquake catalog is a sufficiently complete ground truth for the test periods.
- ad hoc to paper A fixed-length sequence of (magnitude, latitude, longitude, delta_time) is a meaningful representation of seismic history for short-term prediction.
- ad hoc to paper MR, FAR, and ST-MSE adequately measure prediction quality for this task.
Cite this review
Pith. "Pith review of EPBench: A Benchmark for Short-term Earthquake Prediction with Neural Networks." pith.science (2026). https://pith.science/paper/KVN3B33X
@misc{pith2026250515588,
author = {Pith},
title = {Pith review of: EPBench: A Benchmark for Short-term Earthquake Prediction with Neural Networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/KVN3B33X}},
note = {Machine review of arXiv:2505.15588}
}
read the original abstract
Since the beginning of this century, the significant advancements in artificial intelligence and neural networks have offered the potential to bring new transformations to short-term earthquake prediction research. However, currently, there is no widely used benchmark for this task. To address this, we have built a new benchmark (EPBench), which is, to our knowledge, the first global regional-scale short-term earthquake prediction benchmark. Our benchmark comprises 924,472 earthquake records and 2959 multimodal earthquake records collected from seismic networks around the world. Each record includes basic information such as time, longitude and latitude, magnitude, while each multimodal record includes waveform and moment tensor information additionally, covering a time span from 1970 to 2021. To evaluate performance of models on this task, we have established a series of data partitions and evaluation methods tailored to the short-term earthquake prediction task. We also provide a variety of tools to assist future researchers in partitioning the data according to their geographical understanding. Our benchmark includes a variety of neural network models widely used for time series forecasting, as well as a statistical-based model currently employed by seismological bureaus in several countries. We hope this benchmark will serve as a guide to attract more researchers to explore new methods for addressing this task, which holds great significance for human existence. Code is available at https://github.com/CoderZY-X/EPBench
Figures
Figures from the paper (15 more)
Reference graph
Works this paper leans on
-
[1]
Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02556, 2022
arXiv 2022
-
[2]
Neural spatio-temporal point processes
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. arXiv preprint arXiv:2011.04583, 2020
arXiv 2011
-
[3]
Simmtm: A simple pre-training framework for masked time-series modeling
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36:29996–30025, 2023
work page 2023
-
[4]
V Ekambaram, A Jati, NH Nguyen, P Dayama, C Reddy, WM Gifford, and J Kalagnanam. Ttms: Fast multi-level tiny time mixers for improved zero-shot and few-shot forecasting of multivariate time series. arxiv 2024. arXiv preprint arXiv:2401.03955
arXiv 2024
-
[5]
Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting
Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pages 459–469, 2023
work page 2023
-
[6]
Research on history of chinese seismology
Rui Feng and Yuxia Wu. Research on history of chinese seismology. Earthquake Science, 23:243–257, 2010. 9
work page 2010
-
[7]
Rwkv-ts: Beyond traditional recurrent neural network for time series tasks
Haowen Hou and F Richard Yu. Rwkv-ts: Beyond traditional recurrent neural network for time series tasks. arXiv preprint arXiv:2401.09093, 2024
arXiv 2024
-
[8]
Highly accurate protein structure prediction with alphafold
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589, 2021
2021
Show all 25 references
-
[9]
Foundation models for time series analysis: A tutorial and survey
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6555–6565, 2024
2024
-
[10]
Spatial-temporal large language model for traffic prediction
Chenxi Liu, Sun Yang, Qianxiong Xu, Zhishuai Li, Cheng Long, Ziyue Li, and Rui Zhao. Spatial-temporal large language model for traffic prediction. In 2024 25th IEEE International Conference on Mobile Data Management (MDM), pages 31–40. IEEE, 2024
2024
-
[11]
Embracing data incompleteness for better earthquake forecasting
Leila Mizrahi, Shyam Nandan, and Stefan Wiemer. Embracing data incompleteness for better earthquake forecasting. Journal of Geophysical Research: Solid Earth, 126(12):e2021JB022379, 2021
2021
-
[12]
Cred: A deep residual network of convolutional and recurrent units for earthquake signal detection
S Mostafa Mousavi, Weiqiang Zhu, Yixiao Sheng, and Gregory C Beroza. Cred: A deep residual network of convolutional and recurrent units for earthquake signal detection. Scientific reports, 9(1):10267, 2019
2019
-
[13]
The earthquake and tsunami of july 21, 365 ad in the eastern mediterranean sea-review of impact on the ancient world-assessment of recurrence and future impact
George Pararas-Carayannis. The earthquake and tsunami of july 21, 365 ad in the eastern mediterranean sea-review of impact on the ancient world-assessment of recurrence and future impact. Science of Tsunami Hazards, 30(4), 2011
2011
-
[14]
Scalable diffusion models with transformers
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023
2023
-
[15]
Phaselink: A deep learning approach to seismic phase association
Zachary E Ross, Yisong Yue, Men-Andrin Meier, Egill Hauksson, and Thomas H Heaton. Phaselink: A deep learning approach to seismic phase association. Journal of Geophysical Research: Solid Earth, 124(1):856–869, 2019
2019
-
[16]
Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting
Zezhi Shao, Zhao Zhang, Fei Wang, and Yongjun Xu. Pre-training enhanced spatial-temporal graph neural network for multivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pages 1567–1577, 2022
2022
-
[17]
Earthquakenpp: Benchmark datasets for earthquake forecasting with neural point processes
Samuel Stockman, Daniel Lawson, and Maximilian Werner. Earthquakenpp: Benchmark datasets for earthquake forecasting with neural point processes. arXiv preprint arXiv:2410.08226, 2024
2024
-
[18]
Predicting the 1975 haicheng earthquake
Kelin Wang, Qi-Fu Chen, Shihong Sun, and Andong Wang. Predicting the 1975 haicheng earthquake. Bulletin of the seismological society of America, 96(3):757–795, 2006
1975
-
[19]
Earthquake prediction based on spatio-temporal data mining: an lstm network approach
Qianlong Wang, Yifan Guo, Lixing Yu, and Pan Li. Earthquake prediction based on spatio-temporal data mining: an lstm network approach. IEEE Transactions on Emerging Topics in Computing, 8(1):148–158, 2017
2017
-
[20]
Diffstg: Probabilistic spatio-temporal graph forecasting with denoising diffusion models
Haomin Wen, Youfang Lin, Yutong Xia, Huaiyu Wan, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. Diffstg: Probabilistic spatio-temporal graph forecasting with denoising diffusion models. In Proceedings of the 31st ACM International Conference on Advances in Geographic Inform...
2023
-
[21]
Timesnet: Temporal 2d-variation modeling for general time series analysis
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2022
2022 arXiv
-
[22]
https://www.drishtiias.com/to-the-points/paper1/earthquake-4
Yue Wu, Youzuo Lin, Zheng Zhou, David Chas Bolton, Ji Liu, and Paul Johnson. Deepdetect: A cascaded region-based densely connected network for seismic event detection. IEEE Transactions on Geoscience and Remote Sensing, 57(1):62–75, 2018. 10 A Causes of Earthquakes Knowledge f...
2018
-
[23]
Spherical to Cartesian Coordinates Conversion Assume Earth is a perfect sphere with radius R. Convert two points P1(ϕ1, λ1) and P2(ϕ2, λ2) to 3D Cartesian coordinates: x = R cos ϕ cos λ, y = R cos ϕ sin λ, z = R sin ϕ, where: • ϕ: Latitude (positive for north, negative f...
1970
-
[24]
Dropout regularization after each pooling operation mitigates overfitting
Pyramidal Feature Extractor Five consecutive convolution blocks form a contracting pyramidal structure: • Block 1: 512 filters (kernel=7, dilation=5) with padding=15 • Block 2: 256 filters (kernel=5, dilation=4) with padding=8 • Block 3: 128 filters (kernel=5, dilation=3) with...
-
[25]
JPN-IDN" and
Regression Module The regression head comprises two fully-connected layers (128 hidden units) with ReLU non-linearity, mapping the compressed temporal representation to target output. E.2 EP-LSTM The proposed EP-LSTM model implements a deep stacked LSTM architecture with progr...
1996
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.