{"id":"cc98f66e-752f-4190-9c9c-be162e63a4cf","arxiv_id":"2508.10749","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"OpenSWI is a massive open benchmark for surface wave dispersion inversion: 23M+ synthetic pairs, a real-data test set, and a reusable data pipeline.","lead":"Researchers built OpenSWI, a huge benchmark dataset for surface wave dispersion curve inversion, with over 23 million synthetic velocity-dispersion pairs plus real observations. It gives deep learning based seismic imaging a standardized training and evaluation resource, comparable to what ImageNet did for computer vision.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Generalization claim may be circular if OpenSWI-real reference models are conventional inversions of the same dispersion curves.","rationale":"The reader's weakest_assumption explicitly names both forward-model fidelity and reference-model independence. My stress-test focuses on reference-model independence because it is the more directly testable and potentially fatal issue: if references come from the same inversion assumptions that ML is meant to replace, the headline generalization experiment is circular regardless of forward-model details. This does not contradict the reader's UNVERDICTED verdict; it reinforces it. The full text supplied is a different paper (arXiv:2508.10735, SrTiO3), so the actual manuscript's methods, data provenance, and error bars are unavailable. Without access to the SWIDP code and OpenSWI-real metadata, neither the circularity concern nor the forward-model concern can be settled. Thus the appropriate verdict remains UNVERDICTED, and the concrete test above is the minimal check required to move toward a verdict. One additional consideration: if the references are independent (e.g., borehole-constrained), the circularity concern fails, and the remaining forward-model fidelity question becomes secondary; that is why this is the single load-bearing concern.","tokens_in":4036,"tokens_out":3013,"duration_ms":35879,"concrete_test":"Download the SWIDP/open-source metadata for both OpenSWI-real sets. For each reference model, determine its provenance: if the file header/paper states it was derived by inverting the dispersion curve via a code like surf96 or a joint inversion, then the reference is not independent. As a numerical check, take the released observed dispersion curves, run an independent forward-modeling code (e.g., Computer Programs in Seismology) on the released reference models, and compute misfit. If misfit is at the noise level, the references are just forward-model consistent; then compare ML predictions to independent well-log velocity profiles for the same sites, if any exist. If no independent constraints exist, the generalization claim is UNVERDICTED.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Abstract states OpenSWI-real contains 'observed dispersion curves with corresponding reference models.' The demonstration of generalization rests on these references being independent ground truth. Open-source dispersion projects often release phase/group velocity curves along with 1-D velocity models obtained by conventional least-squares inversion (e.g., surf96) of exactly those curves. If the same is true here, then 'strong agreement between predictions and references' only shows that a trained network can imitate that inversion code's mapping. The forward model used to generate the 22M+ synthetic pairs would likely be the same one embedded in such inversion, so the whole evaluation loop is closed: synthetic curves are generated from a layered-model code, references are generated by inverting those curves with the same code, and ML models are trained to invert synthetic curves generated by that code. This does not establish that the dataset is representative of actual Earth structure; it establishes consistency with a particular code's assumptions (fundamental-mode, 1-D flat layers). This is a circularity risk, not an internal inconsistency, and it is precisely the hidden circularity the reader flagged.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OpenSWI, a benchmark dataset for machine-learning inversion of surface-wave dispersion curves. The dataset is generated by a 'Surface Wave Inversion Dataset Preparation (SWIDP)' pipeline and comprises three components: OpenSWI-shallow (over 22 million 1-D velocity/depth profiles paired with fundamental-mode phase and group velocity dispersion curves, derived from the OpenFWI geological models), OpenSWI-deep (1.26 million pairs from 14 global/regional 3-D geological models), and OpenSWI-real (two real-world observed dispersion-curve sets with corresponding reference models). The authors train ML models on the synthetic datasets, evaluate them on OpenSWI-real, and report 'strong agreement' as evidence of the dataset's diversity and representativeness. They release the SWIDP toolbox, datasets, and trained models. The abstract is the only substantive part of the manuscript that matches the claimed arXiv title; the supplied full text is an unrelated condensed-matter paper on SrTiO3.","tokens_in":4163,"tokens_out":4305,"duration_ms":52242,"significance":"If the claims are correct, OpenSWI would fill a genuine gap: a large, standardized, open benchmark for developing and comparing ML methods for surface-wave dispersion inversion. The scale (22M+ shallow and 1.26M deep pairs) and the plan to release code, data, and trained models are significant community contributions. However, the utility of the benchmark rests on two premises that are not established in the available material: (i) that SWIDP's forward modeling faithfully reproduces real dispersion curves, and (ii) that the reference models paired with the real curves are independent ground truth rather than products of a conventional inversion of the same curves. The current abstract-level presentation, with no quantitative evaluation, does not substantiate these premises.","major_comments":[{"comment":"The supplied full text is not the OpenSWI manuscript; it is an unrelated study of quantum phase transitions in SrTiO3 (arXiv:2508.10735). None of the technical sections that would support the abstract are present: the SWIDP pipeline, forward-modeling equations, quality-control thresholds, network architectures, training details, evaluation metrics, or dataset documentation. The central claims of the abstract are therefore unsupported in the reviewed material. This mismatch must be corrected before the paper can be evaluated.","section":"Full text (all sections)"},{"comment":"The utility demonstration rests on the claim of 'strong agreement between predictions and references' on OpenSWI-real. No quantitative metric is given (e.g., RMSE, MAE, R², correlation), no baseline is compared (e.g., conventional inversion or a simple regressor), and no uncertainty estimate is reported. A benchmark dataset paper should include concrete evaluation numbers, per-dataset breakdowns, and ideally a comparison against standard deterministic inversion results. Without these, the generalization claim is not assessable.","section":"Abstract"},{"comment":"The 'corresponding reference models' in OpenSWI-real are the linchpin of the generalization claim. If these reference models were obtained by conventional dispersion-curve inversion of the same observed curves, using a forward code compatible with SWIDP's, then the agreement only demonstrates self-consistency between the synthetic generator and the inversion code—not that the trained models generalize to real Earth structure. The paper must document the provenance of the real reference models (source project, inversion algorithm, forward-modeling assumptions, regularization, and uncertainty) and explicitly demonstrate that they are independent of SWIDP. This is a load-bearing point for the dataset's claimed representativeness.","section":"OpenSWI-real, Abstract"},{"comment":"The claims that OpenSWI-shallow spans 'a wide range of shallow geological structures' and that OpenSWI-deep contains 'high-fidelity' velocity-dispersion pairs are not supported by any quantitative distributional summaries. A benchmark intended for ML generalization should report coverage statistics: velocity and layer-thickness ranges, number of layers, structural families, noise levels, and a comparison of the synthetic input distribution to the real-data distribution. Such statistics are necessary to judge whether the dataset diversity is sufficient for the claimed generalization.","section":"Abstract and dataset descriptions"}],"minor_comments":[{"comment":"The term 'AI-ready' is undefined. Please specify the exact file formats (e.g., HDF5, NumPy, Parquet), data schema, and access/download mechanism, as these are critical for reproducibility and community adoption.","section":"Abstract"},{"comment":"The abstract mentions 'two sets of observed dispersion curves' but gives no metadata: geographic regions, frequency bands, number of stations, or data sources. This information is needed to judge the breadth of the real-world evaluation and to compare with other datasets.","section":"OpenSWI-real, Abstract"},{"comment":"The limitation to 'fundamental-mode phase and group velocity dispersion curves' is an important simplification. The paper should state explicitly that higher modes, off-path propagation, and 3-D heterogeneity are excluded, and discuss how this may affect generalization to real data where these effects are present.","section":"Abstract"},{"comment":"The phrase 'massive-scale' is appealing but the abstract gives only pair counts; file sizes, storage requirements, and the number of independent geological scenarios (rather than pairs) would better characterize the dataset's diversity.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The supplied full text is not the claimed OpenSWI manuscript—it is an unrelated SrTiO3 paper. I would request the correct manuscript before proceeding with review. If the correct manuscript is made available, the authors should be asked to (1) provide quantitative generalization metrics on OpenSWI-real, (2) document the provenance and independence of the real reference models to address the circularity risk, and (3) report distributional diversity statistics for the synthetic datasets. The current abstract alone is insufficient for a soundness judgment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I only had the abstract for this one — the full text attached to the record is actually a SrTiO3 phonon paper (arXiv:2508.10735), not the OpenSWI manuscript. So this read is based on the abstract alone; take it as provisional.\n\nThat said, what's on offer is genuinely useful. A dataset of 22 million shallow and 1.26 million deep synthetic velocity-dispersion pairs, built from OpenFWI models and 14 global/regional 3D models, directly targets a real bottleneck: there is no standard benchmark for deep-learning inversion of surface wave dispersion curves. The SWIDP pipeline, the promised toolbox, and the real-data evaluation set are the right ingredients. If the release matches the abstract, this could be a community resource for years.\n\nThe soft spots are mostly about verification. 'Strong agreement between predictions and references' is stated with no numbers, no baseline, no uncertainty. The real-data references are the bigger question: if the velocity models paired with observed dispersion curves in OpenSWI-real come from conventional inversion code (surf96-style) that uses the same forward modeling as SWIDP, then the generalization test mostly confirms that a trained network can imitate that code's mapping. That's not nothing, but it's weaker than 'representative of real Earth structure.' The stress-test note raises exactly this, and I think it's the right thing to push on. I'm not saying it's circular — just that the abstract doesn't rule it out.\n\nThere's also a mismatch in the review package: the supplied full text isn't this paper. That's a pipeline error, but it means no one can check the forward-modeling details, QC thresholds, or dataset provenance yet. Those are the load-bearing parts.\n\nWho is this for? Anyone working on ML for surface wave dispersion inversion, and benchmark-savvy readers who care about standardized evaluation. It deserves a serious referee, not a desk reject. The referee should ask for actual error bars on the generalization demonstration, a clear statement of how the real-data reference models were constructed, and ideally an independent hold-out check (e.g., against earthquake-based velocity models or borehole data). If the authors can supply that, this is a solid contribution.\n\nMy recommendation: send it out for review, but flag the reference-model independence as the crucial point to verify.","headline":"A potentially valuable benchmark for ML-based surface wave inversion, but the abstract alone can't support the generalization claim and the supplied full text is the wrong paper.","tokens_in":4784,"tokens_out":1498,"would_cite":true,"duration_ms":20394,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Massive synthetic seismic dataset aims to standardize AI-driven surface wave inversion.","keywords":["surface wave dispersion","dispersion curve inversion","machine learning","benchmark dataset","seismic tomography","SWIDP","shallow exploration","deep Earth"],"falsifier":"Train an inversion model on OpenSWI-shallow and -deep, then evaluate it on a held-out real dataset recorded by a different seismic network and processed with an independent dispersion-extraction workflow; if prediction errors on that set are substantially larger than on OpenSWI-real, the dataset's representativeness claim would be falsified. Alternatively, compute dispersion curves from OpenSWI-real's reference models using an independent forward solver and compare them to the observed curves—systematic mismatch would reveal the simplified forward model.","tokens_in":3846,"feed_emoji":"🌊","tokens_out":2963,"duration_ms":36287,"temperature":0.7,"pith_summary":"OpenSWI is a large-scale benchmark dataset for training and testing machine-learning methods that infer underground structure from surface wave dispersion curves. It contains over 22 million shallow and 1.26 million deep synthetic velocity–dispersion pairs, plus two real-world observation sets. The paper's central claim is that this synthetic diversity is sufficient for models trained only on synthetic data to generalize to real observed dispersion data. If true, researchers gain a shared resource to develop, compare, and validate inversion methods without collecting or labeling their own seismic data.","feed_headline":"23 million synthetic curves aim to fix seismic AI inversion","feed_subtitle":"A new benchmark pairs millions of velocity profiles with dispersion curves and real-world tests for generalization.","key_machinery":"The Surface Wave Inversion Dataset Preparation (SWIDP) pipeline, which converts 1-D velocity profiles into paired fundamental-mode phase and group velocity dispersion curves. This forward-modeling machinery generates the large synthetic label set, and the breadth of source geological models determines the structural diversity that is intended to make trained models transfer to real observations.","core_discovery":"The paper claims that the primary obstacle to deep-learning surface wave inversion is the absence of a large, diverse benchmark, not a shortage of model architectures. To remove that obstacle, it introduces OpenSWI, built through the Surface Wave Inversion Dataset Preparation (SWIDP) pipeline. OpenSWI-shallow derives over 22 million 1-D velocity–dispersion pairs from the 2-D OpenFWI geological models; OpenSWI-deep supplies 1.26 million pairs from 14 global and regional 3-D geological models; OpenSWI-real provides observed dispersion curves with reference models for generalization testing. The reported agreement between models trained on the synthetic sets and the real-data references is pres","pith_inferences":["The dataset's structure could transfer to neighboring inverse problems, such as receiver-function inversion or ambient-noise tomography, wherever the input can be represented as dispersion or spectral curves.","The benchmark is built around dispersion-curve misfit; if an application requires full-waveform matching, the synthetic diversity may need to be extended to waveform-level targets.","OpenSWI-real is a single generalization probe; whether the synthetic coverage truly spans the real-data manifold would be tested more rigorously by holding out additional, independently processed real datasets from different regions and networks."],"forward_implications":["Researchers will be able to train and compare machine-learning inversion models on a common benchmark, making results across studies directly comparable.","Models trained on OpenSWI could avoid the strong sensitivity to initial models that plagues traditional dispersion curve inversion.","The released SWIDP toolbox allows the community to generate extended or custom synthetic benchmarks for specialized regions or wave modes.","The two real-data sets provide a standardized test for generalization, letting practitioners benchmark how well synthetic-trained models perform on observed data."],"supporting_citations":[],"fun_headline_variants":["OpenSWI: 23M synthetic curves to train seismic inversion","New benchmark pairs millions of velocity profiles with real curves","Seismic inversion gets a massive AI training ground","22M profiles, 1.26M deep pairs, real tests: OpenSWI","OpenSWI: the dataset that could crack surface wave inversion"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The SWIDP forward model produces dispersion curves that faithfully represent real surface wave dispersion for the modeled structures, and the reference models paired with OpenSWI-real are independent ground truth rather than artifacts of the same inversion assumptions that machine learning is meant to replace.","fun_headline_variants_meta":{"raw":{"variants":["OpenSWI: 23M synthetic curves to train seismic inversion","New benchmark pairs millions of velocity profiles with real curves","Seismic inversion gets a massive AI training ground","22M profiles, 1.26M deep pairs, real tests: OpenSWI","OpenSWI: the dataset that could crack surface wave inversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00044,"raw_usage":{"total_tokens":2129,"prompt_tokens":861,"completion_tokens":1268,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":1180}},"tokens_in":605,"tokens_out":1268,"duration_ms":10841,"temperature":1.0,"reasoning_tokens":1180,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:14:46.212303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an inversion model on OpenSWI-shallow and -deep, then evaluate it on a held-out real dataset recorded by a different seismic network and processed with an independent dispersion-extraction workflow; if prediction errors on that set are substantially larger than on OpenSWI-real, the dataset's representativeness claim would be falsified. Alternatively, compute dispersion curves from OpenSWI-real's reference models using an independent forward solver and compare them to the observed curves—systematic mismatch would reveal the simplified forward model.","supporting_citations":[],"review_version":1}