{"id":"f20e8623-0f33-42bd-b075-5ae25c36e306","arxiv_id":"2412.02780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-modal, multi-scale dataset collection for weather and climate downstream tasks, released with preprocessing code and baseline validations.","lead":"WxC-Bench is a collection of six machine-learning-ready datasets for weather and climate tasks, covering turbulence, gravity waves, precipitation, hurricanes, weather analogs, and forecast text generation. The datasets and preparation code are public, with baseline models shown as sanity checks rather than competitive results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gravity-wave labels rest on an unvalidated spectral-filter definition of 'subgrid fluxes' from ERA5; if the T21 residual is not representative of true GW momentum fluxes, the dataset's central regression task is compromised.","rationale":"The reader's verdict is CONDITIONAL and already flags the GW labels as a weak assumption; my concern singles out the GW label construction as the single most load-bearing issue because it affects the physical meaning of the dataset's central regression task. Unlike the other five tasks, which are mostly based on authoritative sources (HURDAT for hurricane tracks, PIREPs for turbulence, IMERG/PERSIANN-CDR for precipitation), the GW targets are synthesized from an unvalidated spectral filter applied to ERA5 divergent winds. The paper openly acknowledges that ERA5 does not fully resolve gravity waves, and it cites prior work using ERA5, but it does not validate that the T21 residual is a faithful proxy for subgrid momentum fluxes. This is not an internal inconsistency or an ad hominem concern; it is a correctness risk about the scientific meaning of the labels. The paper does provide public artifacts and honest limitation statements, and the technical validation shows the baseline can fit the labels, but internal fit does not establish external validity. The proposed concrete test would settle whether the label definition is robust to the choice of spectral cutoff and whether it represents a physically interpretable flux. If the test shows large sensitivity, the dataset is still useful as a diagnostic benchmark, but the paper's parameterization claim would need to be substantially softened. Since the reader's conditional verdict already anticipates these issues, no verdict change is needed; the condition should explicitly include the GW label validation.","tokens_in":20768,"tokens_out":4938,"duration_ms":57300,"concrete_test":"Recompute the gravity-wave labels for one full year (e.g., 2010) under three alternative definitions: (i) remove the first 42 harmonics instead of 21; (ii) apply a wavelength band-pass retaining only roughly 150-500 km scales; (iii) use ERA5's vorticity and divergence fields without the Helmholtz step. Compare the monthly climatologies and the Attention-Unet baseline R2 to the published numbers. If the climatology shifts by more than about 20% or baseline R2 drops substantially, the label definition is not robust, and the paper must present the GW fluxes as one diagnostic choice rather than as validated subgrid-scale momentum fluxes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is in the 'Gravity Wave (GW) Parameterization' section: the regression targets are computed by Helmholtz-decomposing ERA5 winds into a divergent part, removing the first 21 spherical harmonics, and forming (Fu, Fv) = (udiv * omega', vdiv * omega'). The paper itself calls these 'resolved GW fluxes' and 'subgrid-scale momentum fluxes' in adjacent sentences. WxC-Bench's value for parameterization work depends on these labels being physically meaningful subgrid GW fluxes, not just any divergent residual. ERA5's effective resolution is roughly 150-200 km, so the residual after T21 filtering contains all divergent scales from about 1800 km down to about 150 km; after conservative coarse-graining to 2.8 degrees, much of that signal is actually resolved on the target grid. The 21-harmonic cutoff is arbitrary and unvalidated, and no comparison to independent GW flux estimates (observations, storm-resolving simulations, or other reanalyses) is provided. If the labels are dominated by balanced divergent flow, analysis increments, or truncation artifacts, an ML model trained on them will not learn a useful parameterization, and the flagship task of WxC-Bench is compromised.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WxC-Bench, a collection of six ML-ready datasets for weather and climate downstream tasks: aviation turbulence detection from MERRA-2 and PIREPs, gravity wave momentum flux regression from ERA5, weather analog search over MERRA-2 subgrids, long-range precipitation forecasting from satellite observations, hurricane track and intensity data from HURDAT with MERRA-2 atmospheric states, and natural-language forecast report generation from HRRR and SPC discussion text. For each task, the authors describe the data sources, preprocessing steps, and a baseline model with quantitative or qualitative validation. The datasets and generation code are released publicly on Hugging Face and GitHub.","tokens_in":21034,"tokens_out":5681,"duration_ms":57571,"significance":"If the data curation is sound and the label definitions are revised, WxC-Bench would be a useful multi-modal benchmark resource for weather and climate foundation models. Its strengths include public release of data and code, six tasks spanning meso-β to synoptic scales, and a diversity of ML modalities (classification, regression, retrieval, and captioning). The paper is appropriately cautious in presenting baselines as technical validation rather than state-of-the-art results. The principal value lies in the assembled data and open release, which lower the barrier for future evaluation of generalist weather and climate AI models.","major_comments":[{"comment":"The regression labels are defined as (Fu,Fv) = (udiv * omega', vdiv * omega') from ERA5 divergent winds with the first 21 spherical harmonics removed, then conservatively coarse-grained to 2.8 degrees. The manuscript calls these both 'resolved GW fluxes' and 'subgrid-scale momentum fluxes' in the same paragraph. Because the T21 residual contains divergent scales from roughly 1800 km down to the ERA5 effective resolution (~150-200 km), and the 2.8-degree grid resolves scales above roughly 300 km, the coarse-grained flux is largely a resolved flux rather than a subgrid-scale parameterization target. No independent validation against observations, storm-resolving simulations, or other reanalyses is provided, and the 21-harmonic cutoff is not justified. This is load-bearing for the GW task, which is presented as a central contribution; please add sensitivity analyses to the spectral cutoff, compare against independent GW flux estimates, or explicitly relabel the targets as resolved GW fluxes and remove the subgrid-scale claim.","section":"Gravity Wave (GW) Parameterization, Data Description"},{"comment":"The ML baseline is trained on precipitation estimates from PERSIANN-CDR and IMERG and then evaluated against the IMERG Final product, as the text acknowledges: 'expected given that the precipitation estimates it was trained on were derived from the same precipitation product that is used for the evaluation.' This makes the comparison with the ECMWF/UKMO S2S baselines in Figure 14 circular for the ML model; the claimed lower bias and higher correlation at later lead times are in part an artifact of training on the reference. The dataset can still be useful, but the technical validation should either use a genuinely independent reference (e.g., gauge-based products or a withheld IMERG period) or explicitly frame the ML curve as an in-distribution sanity check rather than a skill comparison.","section":"Long-range Precipitation Forecasting, Figure 14"},{"comment":"Table 7 reports ROUGE-L scores for only three dates (2017-03-30, 2018-08-03, 2019-08-30), with no information on the test-set size, no variance estimates, and no comparison to a trivial baseline. This is insufficient to support the claim that the dataset is ML-ready for end-to-end weather report generation. At minimum, report scores over the full held-out set and include variance or confidence intervals, or restrict the claim to a proof-of-concept.","section":"Technical Validation, Generation of Natural Language-based Weather Forecast Reports, Table 7"}],"minor_comments":[{"comment":"The text states that ERA5 is 'publicly available at a horizontal resolution of 0.3 by 0.3 degrees,' but ERA5 is on a 0.25-degree grid (about 31 km), as the later '30 km' phrase implies; please correct this.","section":"Gravity Wave (GW) Parameterization, Data Description"},{"comment":"The label definition 'any cell with MODG report frequency more than 25%' is an arbitrary threshold; please report sensitivity to this threshold or provide a citation to prior usage.","section":"Aviation Turbulence Prediction, Dataset Description"},{"comment":"The text says that 'the first four images retrieved by the similarity search have overall SSIM scores greater than 0.5, while the fourth and fifth images have scores less than 0.2'; the fourth image cannot satisfy both statements, so please correct the figure or the sentence.","section":"Weather Analog Search, Figure 11"},{"comment":"The phrase 'the dataset comprises of a total of 64x128x24x1461 (~287 million) columns' would be clearer as 'samples' or 'grid-point-time columns'.","section":"Gravity Wave (GW) Parameterization, Data Description"},{"comment":"Since Table 5 appears in the natural-language forecasting section, the first column should specify that the 'Number of samples' refers to HRRR-analysis/report pairs, and the caption should give the full date range and dataset name.","section":"Natural Language-based Weather Forecast Reports, Table 5"},{"comment":"The validation is a single-event case study (Hurricane Michael); please clarify in the text that this is illustrative and not a dataset-level benchmark.","section":"Hurricane Forecasting based on FourCastNet"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest in several places about its limitations (e.g., 'comprehensive evaluation ... is being explored as a future work'), and the open release of data and code is a positive feature. The main concern is the gravity-wave label definition, which is a design choice that needs a direct response rather than a limitation statement. The precipitation baseline comparison should also be reframed to avoid circularity. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my private take on WxC-Bench (2412.02780). If you do any work on multi-task weather or climate models, this is a resource worth knowing: it bundles six heterogeneous tasks into one public benchmark, with code on GitHub and data on Hugging Face. The long-range precipitation input, built from nearly forty years of satellite observations, is genuinely new, and the natural-language forecast pairs are a useful addition that most atmospheric benchmarks lack. The paper is also honest about provenance—it is an extension of WindSet—and about some limitations of the source data.\n\nThe main soft spot is the gravity-wave flux task. The labels are computed by taking the divergent part of ERA5 winds, removing the first 21 spherical harmonics, and calling the residual a subgrid-scale momentum flux. That cutoff is arbitrary, and there is no validation against independent GW estimates (observations, storm-resolving simulations, or other reanalyses). More importantly, after coarse-graining to 2.8 degrees, much of that residual is actually resolved on the target grid, so the 'subgrid' label is a stretch. This does not necessarily destroy the benchmark—the task can still be used to test generalization—but it tempers what the dataset can claim for parameterization work. The authors do flag the resolution limit of ERA5, but they do not address the resolved-versus-subgrid inconsistency.\n\nThe rest of the validations are thin but not damning. The precipitation baseline is trained and evaluated on the same IMERG product family; the authors acknowledge this circularity themselves, but it means the comparison against NWP forecasts is not a fair skill assessment. ROUGE-L is reported on only three dates, the analog search uses a single query, and there are no error bars or repeated seeds. For a dataset paper, these are acceptable as sanity checks, but 'comprehensive evaluation' overstates the evidence.\n\nOn the citation side, the paper leans on prior work from the same group, but it also cites the relevant benchmark literature; nothing looks like a citation violation.\n\nWho is this for? People building or evaluating generalization across weather/climate tasks. It warrants a serious referee—the resource itself is valuable and the flaws are fixable. I would accept it for review and ask for independent validation of the GW labels, a fairer precipitation evaluation, and a more measured description of the baselines.","headline":"Useful public resource of six weather/climate ML tasks with real artifacts; the GW flux labels rest on an unvalidated spectral filter and the validation baselines are uneven, so value the resource but discount the claims.","tokens_in":21581,"tokens_out":3223,"would_cite":true,"duration_ms":32998,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WxC-Bench packages six ML-ready weather and climate datasets, from turbulence labels to forecast text, and validates each with a baseline model.","keywords":["WxC-Bench","weather and climate foundation models","ML-ready dataset","gravity wave parameterization","aviation turbulence prediction","subseasonal precipitation forecasting","hurricane track and intensity prediction","natural-language forecast generation"],"falsifier":"Compare the WxC-Bench gravity wave momentum flux labels to fluxes diagnosed from a storm-resolving global simulation with grid spacing near 2–4 km on the same dates and grid; if the field correlation is near zero or the magnitudes are systematically off, the label construction does not support learning the intended physics.","tokens_in":20598,"feed_emoji":"⛈️","tokens_out":9740,"duration_ms":96352,"temperature":0.7,"pith_summary":"WxC-Bench is a collection of six curated, machine-learning-ready datasets for weather and climate research, each formulated as an ML task: aviation turbulence classification, gravity wave momentum flux regression, weather analog search, subseasonal precipitation forecasting, hurricane track and intensity prediction, and natural-language forecast report generation. The paper's claim is that, together, these datasets span atmospheric scales from meso-$\\beta$ (about $20$--$200$ km) to synoptic scales (about $2500$ km) and multiple data modalities, so they can support training and benchmarking of generalizable weather and climate AI models rather than a single forecasting model. The datasets are released openly with the code used to prepare them, and each is checked with a baseline model in the technical validation section. If the dataset labels are faithful, researchers can use WxC-Bench as a common testbed for foundation-model transfer across tasks that previously had no shared benchmark.","feed_headline":"Six weather and climate datasets, one ML-ready benchmark","feed_subtitle":"Six tasks spanning weather scales, all in ML-ready form for training generalizable forecast models.","key_machinery":"The load-bearing mechanism is the conversion of heterogeneous raw sources into standardized input–label pairs for six ML formulations. For each task, the paper fixes a data source, a preprocessing recipe, and a label construction rule: turbulence labels come from pilot reports binned to MERRA-2 grid cells with more than 25% moderate-or-greater reports; gravity wave fluxes come from a Helmholtz decomposition of ERA5 winds into rotational and divergent parts, removal of the first 21 harmonics, and formation of the products of divergent winds with eddy vertical velocity; analog search uses overlapping 1800 km subgrids of MERRA-2 fields; precipitation uses geostationary, polar-orbiting, and microwave satellite observations regridded to 0.625° by 0.5° with PERSIANN-CDR and IMERG estimates as targets; hurricanes use HURDAT best tracks cubically interpolated to 3-hourly intervals; and forecast reports pair analysis stacks with processed forecast discussions. The pipeline also applies per-task normalization, such as scaling winds by three standard deviations and applying a cube root to gravity wave fluxes, so that each dataset is ML-ready as published.","core_discovery":"The central discovery is that a single publicly released suite of six preprocessed datasets can represent weather and climate phenomena at very different scales and in very different ML formats, and that baseline models can learn from each of them. The six tasks are intended to sample the space of downstream problems: classification of aviation turbulence from MERRA-2 atmospheric profiles, regression of subgrid gravity wave momentum fluxes from ERA5 background states, similarity search over weather analogs encoded as subgrids, autoregressive forecasting of daily precipitation four weeks ahead from four decades of satellite observations, hurricane track and intensity forecasting from HURDAT and MERRA-2, and generation of textual weather discussions conditioned on analysis maps. The validation shows each dataset is learnable: an ANN detects turbulence with 80% overall accuracy, an attention-based convolutional network reproduces global gravity wave flux patterns with $R^2$ up to 0.6 in the midlatitudes, a convolutional encoder-decoder retrieves analogous past weather states, an autoregressive CNN competes with operational subseasonal-to-seasonal models beyond ten-day lead, a Fourier-neural-operator forecast model tracks hurricanes with small track errors, and a vision-language model generates reports that match key forecast wording. The intended consequence is that weather and climate foundation models can be evaluated on one multi-modal, multi-scale benchmark instead of being trained and scored separately for each application.","pith_inferences":["Editorial inference: because the gravity wave labels are computed from ERA5's resolved divergent flow, any systematic deficiency in ERA5's representation of mesoscale gravity waves will be baked into the training targets; comparing these labels against flux estimates from storm-resolving simulations on overlapping dates would quantify that bias.","Editorial inference: the analog-search subgrid encoding, which stores each 1800 km tile with its location, could be reused as a general retrieval interface over other reanalysis products or extended to multi-variable queries, but the paper only demonstrates single-variable lookups.","Editorial inference: the natural-language weather report task is framed as caption generation conditioned on an analysis map, so the benchmark does not yet test whether a model could generate a forecast discussion from a predicted future state; connecting WxC-Bench's forecast tasks to the text labels would close that loop.","Editorial inference: because all tasks are aligned to common reanalysis-era grids and formats, the same pretrained embedding or foundation model could be probed for zero-shot performance across the suite, which would give a cheap signal about where transfer learning fails."],"forward_implications":["A model trained or fine-tuned on WxC-Bench can be scored on six tasks at once, giving a direct measure of transfer across spatial scales and data modalities.","The gravity wave dataset provides a global, multi-year regression target that can be used to train ML parameterizations, which could later be coupled into coarse climate models.","The precipitation benchmark's four-decade record of satellite inputs and precipitation references allows training and evaluation of subseasonal forecasts against operational NWP baselines at lead times beyond ten days.","The hurricane dataset merges Atlantic and Pacific best-track records from 1980 to 2022, so models can be tested for cross-basin generalization rather than only in a single basin.","The natural-language task offers text labels that make it possible to train or fine-tune vision-language models on weather report generation, a step toward automatically communicating forecasts."],"supporting_citations":[{"why":"defines the existing data-driven weather forecasting benchmark that WxC-Bench positions itself against and extends.","marker":"[22]"},{"why":"the prior physics-ML climate emulation dataset that the gravity wave component is compared with and supersedes in scale.","marker":"[26]"},{"why":"the reanalysis source used for aviation turbulence, weather analog, and hurricane inputs.","marker":"[34]"},{"why":"the reanalysis source used to compute gravity wave momentum flux labels and other fields.","marker":"[50]"},{"why":"provides the satellite-based precipitation reference used for the extended training record.","marker":"[70]"},{"why":"provides the precipitation reference used to evaluate the long-range precipitation forecasts.","marker":"[71]"},{"why":"the weather-climate foundation model that WxC-Bench is designed to help benchmark and fine-tune.","marker":"[17]"},{"why":"the earlier dataset suite that this paper extends with additional tasks and detailed evaluation.","marker":"[28]"},{"why":"the Fourier neural operator forecast model used as the hurricane prediction baseline.","marker":"[2]"}],"fun_headline_variants":["Six weather AI tasks, one multi-scale dataset","WxC-Bench: unified ML-ready weather and climate data","From meso to synoptic: one benchmark for weather AI","Multi-modal, multi-scale weather benchmark for AI","All weather scales, six tasks, one dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the residual of ERA5's divergent wind after removing the first 21 harmonics really represents gravity wave momentum fluxes; if that residual is dominated by numerical noise or balanced flow, the gravity wave regression targets are biased.","fun_headline_variants_meta":{"raw":{"variants":["Six weather AI tasks, one multi-scale dataset","WxC-Bench: unified ML-ready weather and climate data","From meso to synoptic: one benchmark for weather AI","Multi-modal, multi-scale weather benchmark for AI","All weather scales, six tasks, one dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000533,"raw_usage":{"total_tokens":2630,"prompt_tokens":1076,"completion_tokens":1554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":1491}},"tokens_in":692,"tokens_out":1554,"duration_ms":12799,"temperature":1.0,"reasoning_tokens":1491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:06:25.585764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the WxC-Bench gravity wave momentum flux labels to fluxes diagnosed from a storm-resolving global simulation with grid spacing near 2–4 km on the same dates and grid; if the field correlation is near zero or the magnitudes are systematically off, the label construction does not support learning the intended physics.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the satellite-based precipitation reference used for the extended training record."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the earlier dataset suite that this paper extends with additional tasks and detailed evaluation."}],"review_version":1}