{"id":"e39d0a79-7f99-4d65-821b-a5ba83744d45","arxiv_id":"2412.00568","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The Well provides 16 diverse physics simulation datasets totaling 15TB of data, with a unified interface and baselines showing standard surrogate models fail on many of the tasks.","lead":"This paper introduces the Well, a 15-terabyte collection of 16 physics simulation datasets with a unified PyTorch interface. It is a benchmark resource for training and evaluating machine learning surrogate models of spatiotemporal dynamics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark physical validity rests on under-resolved, heavily downsampled simulations; the paper provides no quantitative evidence that tasks measure physics rather than solver artifacts.","rationale":"The reader's weakest assumption identifies exactly the concern that is most load-bearing: the benchmark's validity presupposes that the stored simulations are physically faithful enough that models trained on them learn physics. The authors disclose the under-resolution in the datasheet, which is commendable, but disclosure alone does not establish that the learning task is meaningful for surrogate modeling. A concrete resolution check would settle whether the concern actually lands. If the two-resolution datasets show that models trained on the coarser data transfer to the filtered high-resolution data, the concern is mitigated; if not, the benchmark results must be interpreted as measuring coarse-solver emulation rather than physical prediction. Because the paper already received a conditional verdict and the suggested check is a feasible revision rather than a fatal flaw, the appropriate verdict remains conditional; no change from the reader's assessment is needed. I do not elevate the lack of error bars to the primary concern because the baseline experiments are explicitly time-boxed and illustrative, whereas the physical fidelity question bears directly on the core identity of the dataset as a physics simulation collection.","tokens_in":46396,"tokens_out":2617,"duration_ms":30673,"concrete_test":"For MHD, train the FNO baseline on MHD_64 snapshots and evaluate zero-shot on MHD_256 snapshots anti-aliased and downsampled to 64^3. Separately compute the kinetic-energy power spectrum of MHD_64 and of the filtered MHD_256 data; if the spectra diverge substantially below the Nyquist wavenumber, or if the MHD_64-trained model fails to track the filtered high-resolution dynamics, then under-resolution changes the learning target and physical fidelity needs explicit qualification. The same check can be repeated on supernova_explosion_64 versus supernova_explosion_128 to confirm the result across a second dataset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the Well contains 'high quality numerical simulations' (Section 1) and constitutes a benchmark for physical surrogate modeling depends on the data faithfully representing the target dynamics. Appendix A.4 Q13 concedes that 'many of these simulations are under-resolved given the equation parameters used' and interprets this as implicit large-eddy simulation viscosity; Q31 states temporal downsampling 'often occurs by factors upwards of 100.' These admissions are not backed by a quantitative analysis of how much physical content survives. If the stored snapshots are dominated by numerical dissipation, aliasing, or solver-specific discretization errors, then the VRMSE scores in Tables 2 and 3 measure prediction of a particular solver's coarse-grid behavior rather than physical prediction skill. Models trained on such data may learn to imitate numerical viscosity instead of the governing equations, undermining the stated goal of accelerating simulation-based workflows. The paper's assertion that under-resolution 'does not affect the majority of use cases' is plausible but unsupported; the two-resolution datasets (MHD_64 from MHD_256, supernova_explosion_64/128) provide a natural opportunity to test this, and the paper does not use them for that purpose.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Well, a 15 TB collection of 16 (per the abstract) physics simulation datasets intended to support machine-learning surrogate modeling of spatiotemporal physical systems. The datasets cover a wide range of problems, from acoustic scattering and active matter to magnetohydrodynamics, supernova explosions, and viscoelastic instabilities, and are stored in a unified HDF5 format with a provided PyTorch interface. The authors present baseline results with FNO, TFNO, U-net, and CNextU-net on one-step prediction and autoregressive rollout tasks, using variance-scaled RMSE (VRMSE) as the primary metric. The paper emphasizes diversity, scale, and the difficulty of the resulting benchmark tasks, noting that several baseline models fail to beat a constant-mean predictor.","tokens_in":46583,"tokens_out":5517,"duration_ms":65764,"significance":"If the underlying simulations faithfully represent the target physics, the Well would be a valuable community resource: it combines scale, diversity, domain-expert involvement, open code and data distribution, a self-documenting data specification, and a flexible benchmarking library. The inclusion of two-resolution versions of MHD and supernova simulations, the use of standard external model architectures, and the transparent reporting of hyperparameters and data-generation details are notable strengths. The benchmark results convincingly show that generic off-the-shelf models struggle on many of these tasks, which supports the paper's motivating claim that more challenging and diverse datasets are needed. However, the paper's central claim that these are 'high quality numerical simulations' suitable for benchmarking physical surrogate models is not yet fully supported, because the effects of under-resolution and aggressive temporal downsampling on physical fidelity are not quantified.","major_comments":[{"comment":"The paper's central claim that the Well contains 'high quality numerical simulations' and that the VRMSE scores in Tables 2 and 3 measure physical surrogate-model skill rests on the assumption that the stored snapshots faithfully represent the target dynamics. Appendix A.4 Q13 concedes that 'many of these simulations are under-resolved given the equation parameters used' and frames this as implicit large-eddy-simulation viscosity, while Q31 states that temporal downsampling 'often occurs by factors upwards of 100.' These admissions are not accompanied by any quantitative evidence—such as spectral convergence checks, spatial resolution studies, or comparisons between the low-resolution stored data and higher-resolution references—that the retained fields still capture the physically relevant behavior. The two-resolution datasets (MHD_64/MHD_256 and supernova_explosion_64/128) provide a natural control for exactly this question, but the paper does not use them for that purpose. I request a quantitative resolution-fidelity analysis, or a clear and prominent restatement of the benchmark's scope as measuring prediction of a particular solver's coarse-grid dynamics rather than of the underlying physical system.","section":"Appendix A.4 (Q13, Q31); Section 1; Section 4"},{"comment":"Several reported benchmark results come from models that saw fewer than five epochs within the 12-hour compute budget; for example, Table 6 shows CNextU-net at 1 epoch on euler_multi_quadrants, 3 epochs on turbulence_gravity_cooling, and 3 epochs on turbulent_radiative_layer_3D, and FNO/TFNO at 4 epochs on euler_multi_quadrants. The checklist explicitly states that no error bars are reported. Without seed variance or at least an explicit marker for under-trained runs, the model-comparison claims in Section 4 (e.g., '9/17 favor U-net type models while 8 favor spectral') are not robust, and close entries in Table 2 (e.g., acoustic_scattering, MHD_64, turbulent_radiative_layer_3D) could change with additional training or random seeds. Please report multiple random seeds for at least the contested datasets, or clearly flag which results are limited by the time budget and refrain from architectural conclusions based on those entries.","section":"Section E.1, Table 6, Checklist item 3(c)"},{"comment":"The one-step results in Table 2 and the rollout results in Table 3 use different evaluation protocols (sliding windows from ground truth vs. rollouts initiated from the beginning of the simulation), and the paper explains that the two settings can therefore disagree. However, the claim that 'loss sometimes decreases in later windows' due to dissipative physics is not supported by the aggregated window averages in Table 3 alone. I recommend showing per-dataset windowed VRMSE curves, ideally with confidence intervals, so that the reader can distinguish physical dissipation from trajectory-dependent variability; alternatively, state explicitly that the current table does not allow such a distinction.","section":"Section 4, Tables 2 and 3"}],"minor_comments":[{"comment":"Table 1 contains 17 rows (counting MHD and supernova_explosion as single rows with two resolutions each), while the abstract states '16 datasets.' Please clarify whether MHD_64/MHD_256 and supernova_explosion_64/128 are each counted as one dataset or two, and make the numbering consistent throughout.","section":"Table 1 and Section 3.1.7/3.2"},{"comment":"The phrase 'reddening glow called akilonova' should be corrected to 'a kilonova.'","section":"Section 3.1.9"},{"comment":"The limitations discussion in the main text (Section 5) does not mention the under-resolution and temporal-downsampling caveats that appear only in the appendix. Since these are central to the benchmark's interpretation, they should be stated, at least briefly, in the main text.","section":"Section 4 and Appendix A.4 Q13"}],"recommendation":"major_revision","confidential_remarks":"This is a potentially useful dataset release, and the authors have been unusually transparent about generation details and limitations. The main risk is the physical-fidelity question: if the heavily downsampled, under-resolved snapshots are dominated by solver artifacts, the benchmark's value as a physics benchmark is diminished. The two-resolution datasets are a natural way to test this, and I would like to see that analysis before acceptance. The missing error bars and under-trained baseline runs are a second, more standard issue that should be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, useful dataset release, and it deserves a serious referee. The value is infrastructural rather than scientific: 15 TB, 16 datasets, a common HDF5 spec, a clean PyTorch interface, and a remarkably candid datasheet. The authors generated the data with established solvers, and they ship code and generation scripts where possible. That is real, citable work.\n\nWhat is actually new is the aggregation and unification. Prior benchmarks were either small and simple (PDEBench-style) or large and single-domain (ClimSim, BLASTNet). The Well is the first at this scale to combine research-frontier simulations from multiple fields with a single interface and baseline harness. The domain-expert involvement shows in the dataset choice and metadata.\n\nThe soft spots are real but not fatal. First, the stress-test concern lands: Appendix A.4 Q13 admits many simulations are under-resolved, and Q31 says temporal downsampling often exceeds 100x. The introduction calls the data \"high quality numerical simulations,\" and the benchmark section implies tasks measure physical prediction. But there is no quantitative analysis of how much physical content survives. The natural control exists—MHD_64 was downsampled from MHD_256, and supernova_explosion has two resolutions—and the paper does not use it to check whether models learn physics or solver artifacts. This does not sink the paper, because for many ML applications, learning a solver's coarse-grid behavior is still useful. But the claim \"does not affect the majority of use cases\" is an assertion, not a demonstration.\n\nSecond, the baselines are underpowered. No error bars (checklist admits), and four datasets get fewer than 5 training epochs within the 12-hour budget. The paper is upfront that these are example baselines, so this is a limitation rather than a flaw. Still, Tables 2 and 3 should not be read as reliable model rankings; close calls are noise.\n\nWho is this for? Researchers building or evaluating surrogate models who need a diverse benchmark suite. It is the most useful public option at this scale. I would bring it to reading group and I would cite it. A revision should add a resolution-sensitivity analysis (using the paired datasets) and error bars for at least a few representative runs. With those, this becomes a standard reference.","headline":"A genuinely useful dataset collection worth having, but the paper's physical-fidelity claims outrun the evidence and the baselines are too underpowered to rank models.","tokens_in":47239,"tokens_out":2382,"would_cite":true,"duration_ms":24507,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Well contributes 15TB of physics simulation data and shows standard surrogate models often cannot beat a constant-mean predictor.","keywords":["physics simulation datasets","surrogate modeling","benchmark suite","spatiotemporal prediction","neural operators","partial differential equations","dataset release","variance-scaled RMSE"],"falsifier":"Train a competitive model on each of the 16 datasets without the 12-hour cap and with per-field loss normalization; if test VRMSE falls below 1 on nearly all datasets while the paper's reported baselines stay above it, the claim that these tasks challenge current surrogate models would be weakened.","tokens_in":46193,"feed_emoji":"🌌","tokens_out":8087,"duration_ms":75541,"temperature":0.7,"pith_summary":"The paper introduces the Well, a 15-terabyte collection of 16 numerical simulation datasets spanning a wide range of spatiotemporal physics, from acoustic scattering and reaction-diffusion to magnetohydrodynamic turbulence and supernova explosions. It argues that existing physics-oriented machine-learning benchmarks cover too narrow a class of behavior, and that this collection offers the volume, diversity, and difficulty needed to evaluate surrogate models and to push toward general-purpose physics models. To support that claim, the paper reports baseline experiments: four standard surrogate architectures trained with a modest compute budget frequently score above 1 on a variance-scaled error, meaning a constant-mean prediction would have been better, and longer autoregressive rollouts often score above 10. The release also includes a common file format and a unified Python interface, so the datasets can be used individually or as a benchmark suite.","feed_headline":"A 15TB physics simulation suite stresses AI surrogates","feed_subtitle":"Off-the-shelf neural models often can't beat a constant-mean predictor on the Well's 16 datasets.","key_machinery":"The defining object is the collection itself, organized by a shared self-documenting archive specification that stores state fields as arrays indexed by trajectory, time, and spatial coordinates, together with metadata for physical parameters and boundary conditions, and read through a uniform Python dataset loader. The benchmark's organizing metric is variance-scaled root mean squared error, $\\mathrm{VRMSE}(u,v)=\\sqrt{\\langle |u-v|^2\\rangle / \\langle |v-\\bar v|^2\\rangle}$, so a score of 1 means the model does no better than predicting the spatial mean and scores above 1 mean worse than that trivial predictor. The working task in the reported baselines is autoregressive next-snapshot prediction from a four-step history, with trajectories split 80/10/10 into training, validation, and test.","core_discovery":"The central claim is that the Well provides a benchmark collection that is simultaneously large, diverse, and demanding: 16 datasets totaling 15TB of temporally coarsened snapshots from expert simulations, stored on uniform grids under a shared self-documenting format and accessed through a common Python interface. In the paper's own baseline experiments, off-the-shelf architectures trained for about 12 hours on one GPU fail to beat a constant-spatial-mean predictor on several one-step tasks and on most longer rollouts; for example, the rayleigh_taylor_instability dataset yields a variance-scaled RMSE above 10 for all four tested models. The authors take this as evidence that the suite poses new challenges that will inform the next generation of data-driven surrogates, and they emphasize that the baselines are deliberately simplistic rather than tuned peak performance.","pith_inferences":["A natural test the paper does not run is generalization across the included parameter ranges, such as training on some Gray-Scott (f,k) pattern families and testing on held-out ones; this would directly probe whether the suite measures physical understanding or only interpolation.","Because the authors concede that many simulations are under-resolved and that temporal downsampling often exceeds a factor of 100, part of the measured difficulty may be solver-specific; re-running one dataset at higher resolution or with denser snapshots would separate learned physics from learned artifacts.","The VRMSE normalization centers on the spatial mean of each field, so the same model error can look very different for fields whose mean is near zero versus bounded away from zero; comparing with uncentered normalization could change model rankings.","The collection's uniform schema makes it a plausible testbed for multi-physics pretraining and transfer, a direction the paper motivates in passing but does not evaluate."],"forward_implications":["Several one-step tasks and most longer rollouts beat the constant-mean predictor, with VRMSE above 10 on rayleigh_taylor_instability and on later windows of rayleigh_benard and shear_flow; new methods are needed for these regimes.","No architecture class dominates: U-net-style models win 9 of 17 one-step experiments and spectral models win 8, suggesting one-model-fits-all surrogate approaches may struggle.","One-step accuracy does not guarantee rollout accuracy: time-averaged losses over windows 13-30 are often far worse, and sometimes decrease because dissipative systems become smoother.","The common format and interface let third-party datasets be added without modification to the code base.","Beyond the forward problem, several datasets are positioned for inverse scattering, super-resolution, cross-dimensional transfer, parameter-range generalization, and long-term stability studies."],"supporting_citations":[{"why":"Defines the Fourier neural operator architecture used as a benchmark baseline.","marker":"[97]"},{"why":"Defines the U-net architecture used as a benchmark baseline.","marker":"[99]"},{"why":"Numerical solver used to generate the acoustic scattering and Euler multi-quadrant datasets.","marker":"[110]"},{"why":"Numerical solver used for the red supergiant convection and turbulent radiative layer datasets.","marker":"[111]"},{"why":"Spectral framework used for several datasets, including shallow water, Rayleigh-Bénard, shear flow, and viscoelastic instability.","marker":"[112]"},{"why":"Radiation GRMHD code used to generate the post-merger accretion disk simulations.","marker":"[113]"},{"why":"Provides the turbulent radiative mixing layer simulations included in the Well.","marker":"[95]"},{"why":"Provides the viscoelastic instability simulations of coexisting attractors and edge states.","marker":"[96]"}],"fun_headline_variants":["AI surrogates flounder on 15TB physics benchmark","Physics suite challenges ML: simple baselines beat neural nets","Massive physics dataset exposes AI simulation limits","The Well: 15TB of physics simulations stumps off-the-shelf AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's difficulty claims rest on the assumption that the stored snapshots, downsampled by factors often above 100 and sometimes under-resolved, still faithfully represent the target physics, so that models trained on the Well learn physics rather than solver artifacts.","fun_headline_variants_meta":{"raw":{"variants":["AI surrogates flounder on 15TB physics benchmark","Physics suite challenges ML: simple baselines beat neural nets","Massive physics dataset exposes AI simulation limits","The Well: 15TB of physics simulations stumps off-the-shelf AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2629,"prompt_tokens":896,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1663}},"tokens_in":512,"tokens_out":1733,"duration_ms":73163,"temperature":1.0,"reasoning_tokens":1663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:12:03.149537+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a competitive model on each of the 16 datasets without the 12-hour cap and with per-field loss normalization; if test VRMSE falls below 1 on nearly all datasets while the paper's reported baselines stay above it, the claim that these tasks challenge current surrogate models would be weakened.","supporting_citations":[],"review_version":1}