{"id":"ae798375-7dee-4e43-81a7-0090efb0b173","arxiv_id":"2505.04835","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Synthetic corruptions are a strong proxy for real-world corruptions when ranking semantic segmentation models on average, but individual corruption types like fog and night show weak correlation.","lead":"This study tests whether computer-vision models' performance on synthetic image corruptions, like added fog or snow, predicts their performance on real photos taken in bad weather. Across dozens of semantic segmentation models, average scores on synthetic corruptions strongly correlate with scores on real adverse-condition photos, suggesting synthetic tests can stand in for expensive real-world data collection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The r≈0.8 correlation is not shown to be corruption-specific; without controlling for general model quality (clean mIoU, parameter count) or using a rank-based statistic, synthetic corruptions may predict ACDC only through generic capacity, undercutting the Sec. 4.2 corruption-specific conclusions.","rationale":"I read this in good faith: the study assembles a large set of Cityscapes-trained segmentation models, evaluates them on Common Corruptions and ACDC, and reports the cross-model correlations; code is promised open-source and the appendix limitations paragraph (F.1) is honest about the restricted number of settings benchmarked. The observed correlation of roughly 0.76-0.80 is visible in the scatter plots and is not in dispute. My concern is that the interpretation is under-determined. The paper's own Fig. 8 shows most per-corruption mIoU values rising with parameter count and notes duplicate points from two crop sizes, so the scatter's non-independence and capacity trend are documented in the appendix; under those conditions a Pearson correlation near 0.8 is expected even if synthetic corruptions carry no corruption-specific information. The missing control is decisive: whether clean Cityscapes mIoU or parameter count alone predicts ACDC mIoU as well as the synthetic benchmark does. If it does, the strong reading ('we do not need to go into the wild'; snow r=0.867 as evidence that synthetic snow mimics real snow) is not supported, while the weak relative-ranking reading likely survives. This is exactly why I do not move to reject: the literal claim is about relative analysis and may hold. But the paper must supply the capacity control or narrow its claims, and it must explain the Fig. 2 vs Fig. 3 discrepancy (0.795 vs 0.759 for the same scatter). This agrees with the reader's weakest assumption, and the CONDITIONAL verdict should stand with these analyses as explicit conditions.","tokens_in":14598,"tokens_out":11531,"duration_ms":109499,"concrete_test":"Recompute, for every Cityscapes-trained model in Table 1, the clean validation mIoU, mean synthetic-corruption mIoU, per-condition ACDC mIoU, and log parameter count. Then: (1) report Spearman rho and leave-one-out sensitivity for mean synthetic vs ACDC mIoU; (2) compute the partial Pearson correlation of synthetic vs ACDC controlling for clean mIoU, and separately for log parameter count; (3) compare the clean-vs-ACDC correlation with the synthetic-vs-ACDC correlation. If the partial correlation falls below about 0.3, or if clean performance predicts ACDC as well as synthetic corruptions do, the claim must be weakened to 'synthetic corruptions rank models only to the extent that any test set would.' Also build the full 15-corruption x 4-condition correlation matrix and test, after capacity control, whether matched pairs (snow-snow, fog-fog) exceed unmatched pairs; if not, the Sec.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Sec. 4.1: 'for relative analysis that synthetic corruptions do serve as a reliable proxy for real-world corruptions') rests on a cross-model Pearson correlation of about 0.8 between mean 2D Common Corruption mIoU (severity 3) and ACDC mIoU. For that claim to license the paper's practical conclusions ('we do not need to go into the wild'; 'supporting the use of synthetic corruptions for robustness evaluation'), the correlation must reflect corruption-specific signal rather than the fact that better models are better on every test set. The paper never runs this control: it reports no comparison of the synthetic-vs-ACDC correlation against (i) clean Cityscapes validation mIoU vs ACDC, or (ii) log parameter count vs ACDC. The appendix makes the confound plausible: Fig. 8 shows mIoU on most individual corruptions rising with parameter count, and the Fig. 8 caption notes that the same architecture/backbone pair appears twice (two crop sizes), so the scatter points are not independent. If clean in-domain performance or parameter count predicts ACDC as well as the synthetic benchmark does, the observed r≈0.8 is a generic-capacity artifact, and synthetic corruptions add no corruption-specific predictive signal. The weak relative-ranking statement might survive, but Sec. 4.2's corruption-specific conclusions would not: the claimed snow mimicry (r=0.867), and the claimed failures of brightness (r=0.270) and fog (r=0.349), would each be capacity-contaminated. Two compounding issues: the identical scatter in Fig. 2 and Fig. 3 reports different correlations (0.795 vs 0.759) without explanation, and a claim about relative analysis is argued with Pearson, an outlier-sensitive linear statistic, rather than a rank-based measure such as Spearman's rho.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks a large collection of semantic segmentation models trained on Cityscapes by evaluating them on the real-world ACDC dataset and on 2D Common Corruptions applied to Cityscapes at severity 3. It reports Pearson correlations between mean mIoU on synthetic corruptions and mIoU on ACDC, finding values around 0.8, and interprets this as evidence that synthetic corruptions are a reliable proxy for real-world robustness for relative model comparisons. The paper also reports corruption-specific correlations (snow 0.867, brightness 0.270, fog 0.349, frost vs. fog 0.680) and provides benchmarking results on ADE20K and PASCAL VOC. The central conclusion is that synthetic corruptions support robustness evaluation, while individual corruption types are not universally reliable proxies.","tokens_in":14900,"tokens_out":5576,"duration_ms":51628,"significance":"If the reported correlation is genuinely corruption-specific rather than an artifact of general model quality, the finding is practically valuable: it would justify using cheap synthetic corruptions for model selection and robustness screening in semantic segmentation. The benchmark itself, spanning many architectures and backbones, is a useful community resource. However, the key statistical claim currently lacks controls for clean performance and model capacity, does not report confidence intervals or the number of models used, and contains an unresolved discrepancy in the headline correlation. The significance of the paper therefore hinges on additional analysis that the manuscript does not provide.","major_comments":[{"comment":"The same comparison, mean 2D Common Corruption mIoU versus ACDC mIoU, is reported with Pearson correlation 0.795 in Fig. 2 and 0.759 in Fig. 3. Because this is the headline evidence for the paper's central claim, the discrepancy must be resolved, and the exact number and identity of models used in each scatter plot should be stated. Without a consistent model set, the reproducibility of the main correlation is in question.","section":"Sec. 4.1, Figs. 2 and 3"},{"comment":"The r approximately 0.8 correlation is not shown to be corruption-specific. Appendix Fig. 8 shows that mIoU on most individual corruptions rises with parameter count, and its caption acknowledges that the same architecture/backbone pair appears twice due to two different crop sizes; these duplicated, non-independent points may inflate the reported correlations. The paper should control for general model quality, for example by reporting partial correlations with clean Cityscapes validation mIoU and log parameter count, or by comparing the synthetic-vs-ACDC correlation with the clean-vs-ACDC correlation. Without such a control, the corruption-specific conclusions of Sec. 4.2, such as snow at 0.867 versus brightness at 0.270, may reflect general capacity rather than a shared corruption-specific signal.","section":"Sec. 4.1 and Appendix Fig. 8"},{"comment":"The corruption-specific Pearson correlations are reported without confidence intervals, p-values, or the number of models used in each analysis. With an unreported sample size and multiple comparisons across corruption types, the qualitative claims of 'very strong alignment' for snow and 'weak alignment' for brightness and fog are not statistically supported. The authors should provide uncertainty quantification or a rank-based statistic, and should also justify the use of Pearson r if the scatter plots reveal nonlinear monotone relationships.","section":"Sec. 4.2"},{"comment":"The paper's central claim is stated for 'real-world corruptions' in general, but the only real-world dataset used is ACDC, which the related-work section itself describes as covering only a limited set of conditions 'to a certain extent'. The title and abstract should be qualified to 'adverse weather and lighting conditions' or the paper should include additional real-world datasets before making the broader generalization.","section":"Title, abstract, and conclusion"}],"minor_comments":[{"comment":"The sentence 'we prove this assumption' is too strong for an empirical correlation analysis; suggest using 'provide evidence for' or 'test'.","section":"Sec. 4.1"},{"comment":"The main text should clarify that Figs. 2 and 3 use only the Cityscapes-trained subset of Table 1, and should report the resulting sample size n in each panel.","section":"Sec. 4.1 and Table 1"},{"comment":"The duplicate architecture/backbone pairs caused by two crop sizes are disclosed in the appendix, but this non-independence should also be mentioned in the main-text discussion of Figs. 2 and 3, and markers should distinguish the two crop sizes.","section":"Appendix Fig. 8 caption"},{"comment":"Appendix A reports correlation ranges such as 0.782-0.858 without stating what the range spans; please specify the exact comparisons and model sets used to produce each number.","section":"Appendix A"},{"comment":"The limitations paragraph acknowledges the lack of severity-level analysis, but it does not address the capacity confound identified above; a brief sentence acknowledging this would help calibrate the reader's expectations.","section":"Appendix F.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a compressed workshop-style paper submitted to a journal; the main text is very short and relies heavily on appendix material. The reference list contains many self-citations that are not all essential to the presented claims; I would ask the editor to assess whether the citation pattern is appropriate. The core empirical question is interesting, but the statistical analysis needs the capacity-control experiments and uncertainty quantification described in the major comments before the central claim is fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, large-scale empirical study, but the headline claim is not yet backed by the right analysis. The benchmark (multiple architectures/backbones on Cityscapes-C severity 3 and ACDC) is a real contribution; the snow vs. brightness/fog split is a nice, non-obvious finding. But the paper's central evidence is a single cross-model Pearson correlation on mean mIoU, and it never checks whether that correlation is anything more than 'better models are better on every test set.' That is the gap that matters.\n\nWhat is new: first large-scale comparison of semantic segmentation models on 2D Common Corruptions vs. ACDC. The authors also provide per-corruption correlations and a GAM worst-case measure. The appendix benchmarking on ADE20K and PASCAL VOC is a useful resource. The code is promised open-source. Good.\n\nSoft spots, in order.\n\n1. No control for general model quality. The paper reports r≈0.8 between mean synthetic mIoU and ACDC mIoU across models. The appendix's own Fig 8 shows mIoU on most corruptions rises with parameter count. Without showing that synthetic corruptions predict ACDC better than clean Cityscapes mIoU or parameter count does, the r≈0.8 could be a capacity artifact. The claim 'we do not need to go into the wild' would not follow. This also contaminates the corruption-specific correlations: snow's 0.867 may just track model size, and fog's 0.349 may be low because that corruption is less capacity-sensitive. The paper needs partial correlations or a rank-based comparison against clean performance.\n\n2. Unreported statistics and an unexplained inconsistency. No n, no confidence interval, no p-value. Fig 2 reports Pearson 0.795 for mean-vs-ACDC; Fig 3 reports 0.759 for what looks like the same scatter. The captions don't explain the difference. That is exactly the kind of thing a referee will catch.\n\n3. The word 'prove' in Sec 4.1 overstates a correlational result. Minor but symptomatic.\n\n4. Non-independence: the same architecture/backbone pair appears at two crop sizes, so scatter points are not independent. A mixed model or at least a stated unique-model count would help.\n\nI don't see a circularity problem; the analysis is empirical and the GAM is just a definition. The ACDC-reference limitation is real but acknowledged implicitly by the dataset choice; it slightly limits the 'real-world' generalization, but not fatal.\n\nWho this is for: anyone building robustness benchmarks for driving perception, and anyone who wants to know whether synthetic corruptions can substitute for ACDC-style collection. The paper deserves a serious referee, but on the current evidence I would accept with major revision, not as is.","headline":"The benchmark is genuinely useful, but the paper's central correlation claim lacks the controls needed to separate corruption-specific signal from generic model capacity.","tokens_in":15506,"tokens_out":2862,"would_cite":false,"duration_ms":26270,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that, for relative robustness evaluation, synthetic corruptions rank semantic segmentation models nearly the same as real-world adverse conditions.","keywords":["synthetic corruptions","real-world corruptions","robustness evaluation","semantic segmentation","distribution shift","ACDC","Common Corruptions","Pearson correlation"],"falsifier":"Compute the partial correlation between mean synthetic-corruption mIoU and ACDC mIoU after controlling for clean Cityscapes mIoU or model parameter count; if the partial correlation falls to near zero, the apparent proxy signal is mostly general model quality rather than corruption-specific robustness. Alternatively, test a set of models matched for clean accuracy but known to differ in corruption robustness: if their synthetic and real rankings diverge, the proxy claim fails.","tokens_in":14390,"feed_emoji":"🌫️","tokens_out":6404,"duration_ms":57949,"temperature":0.7,"pith_summary":"This paper asks whether synthetic image corruptions, which are cheap to generate and widely used, can take the place of real-world adverse-condition data when evaluating how robust a semantic segmentation model is. It benchmarks a large set of models trained on the Cityscapes dataset against both 2D Common Corruptions and the ACDC real-world adverse-conditions dataset, and compares the two orderings of models. The central finding is a Pearson correlation of about 0.795 between mean performance on synthetic corruptions at severity 3 and performance on real-world corruptions, with an even higher correlation of 0.828 for worst-case synthetic performance. The paper therefore concludes that, for relative analysis, ranking or selecting among models, synthetic corruptions are a reliable proxy for real-world robustness, even though individual corruption types vary in how well they align.","feed_headline":"Synthetic corruptions rank segmentation models like real-world conditions","feed_subtitle":"Across many Cityscapes-trained models, mIoU on synthetic and real adverse conditions correlates near 0.8.","key_machinery":"The central object is a correlation benchmark pairing two test suites: synthetic 2D Common Corruptions, namely the 15 perturbation types applied at severity 3 to Cityscapes validation images, and ACDC, a dataset of images captured in real adverse conditions. The load-bearing comparison is the Pearson correlation between models' mean mIoU on the two suites, complemented by GAM3, the worst mIoU across all synthetic corruptions at severity 3, which captures the model's weakest synthetic condition. This design lets the paper ask both aggregate questions about whether the model ranking transfers and corruption-specific questions about which synthetic perturbations predict which real conditions.","core_discovery":"The paper's claim is that, for relative analysis, synthetic corruptions do serve as a reliable proxy for real-world corruptions in semantic segmentation. The evidence is a large benchmarking study: models trained on Cityscapes and spanning many architectures and backbones are evaluated on the 15 synthetic 2D Common Corruptions at severity 3 and on the real-world ACDC dataset under night, fog, rain, and snow. Mean mIoU across synthetic corruptions correlates with ACDC mIoU at a Pearson correlation of about 0.795, and the worst-case synthetic measure GAM3 correlates at 0.828, so the ordering of models by synthetic robustness closely matches their ordering by real-world robustness. Corruption-specific analysis finds that synthetic snow aligns strongly with real snow at 0.867, frost correlates moderately with real fog at 0.680, while synthetic brightness versus real night at 0.270 and synthetic fog versus real fog at 0.349 align weakly. The paper reads this as evidence that synthetic corruptions work as a proxy at the level of aggregate model ranking, while specific corruption types need refinement.","pith_inferences":["Beyond the paper: the aggregate correlation might be partly explained by general model quality, since better models tend to score higher on every test set; a partial-correlation control for clean Cityscapes mIoU or parameter count would show whether the synthetic-to-real link survives once general quality is removed.","Beyond the paper: the weak fog correlation suggests a concrete improvement path, namely physically based fog simulation or depth-aware rendering, rather than the current 2D synthetic fog, before synthetic fog can rank models for real fog.","Beyond the paper: a practitioner could use synthetic corruptions to pre-screen a large model zoo and then validate only the top candidates on real-world data; the paper's relative-analysis framing supports this two-stage workflow, though it tests no such protocol."],"forward_implications":["If the correlation holds, robustness evaluation for semantic segmentation can be run largely on synthetic corruptions, avoiding the cost of collecting real-world adverse-condition images.","The worst-case synthetic measure GAM3 predicts real-world performance at least as well as mean synthetic performance, so a robustness test built around the model's weakest corruption is a sensible design choice.","Synthetic snow is a trustworthy stand-in for real snow, while synthetic brightness and fog are not; benchmarks targeting nighttime or foggy driving should supplement synthetic tests with real data.","The released benchmarking numbers for Cityscapes, ADE20K, and PASCAL VOC2012 give practitioners reference points for comparing future models on synthetic corruptions.","The claim is explicitly about relative analysis; absolute mIoU on synthetic data should not be read as the expected absolute mIoU in the field."],"supporting_citations":[{"why":"Defines the 15 2D Common Corruptions that form the synthetic corruption test set.","marker":"[31]"},{"why":"Provides the ACDC real-world adverse-conditions dataset used as the real-world reference.","marker":"[50]"},{"why":"Supplies the Cityscapes dataset on which models are trained and on which synthetic corruptions are applied.","marker":"[15]"},{"why":"Establishes the Cityscapes-C protocol and the choice of corruption severity 3.","marker":"[46]"},{"why":"Prior robustness benchmark for semantic segmentation that this study extends to more architectures and to real-world correlations.","marker":"[35]"}],"fun_headline_variants":["Synthetic corruptions predict real-world model ordering","Synthetic vs real corruptions: strong correlation in rankings","Synthetic benchmarks mirror real-world robustness ranks","Synthetic corruptions: reliable proxy for real-world testing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that ACDC stands in for real-world corruptions generally and that the correlation across models reflects shared corruption-robustness rather than simply better models scoring higher on every test set.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic corruptions predict real-world model ordering","Synthetic vs real corruptions: strong correlation in rankings","Synthetic benchmarks mirror real-world robustness ranks","Synthetic corruptions: reliable proxy for real-world testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1705,"prompt_tokens":923,"completion_tokens":782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":720}},"tokens_in":539,"tokens_out":782,"duration_ms":8226,"temperature":1.0,"reasoning_tokens":720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:20:04.106229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the partial correlation between mean synthetic-corruption mIoU and ACDC mIoU after controlling for clean Cityscapes mIoU or model parameter count; if the partial correlation falls to near zero, the apparent proxy signal is mostly general model quality rather than corruption-specific robustness. Alternatively, test a set of models matched for clean accuracy but known to differ in corruption robustness: if their synthetic and real rankings diverge, the proxy claim fails.","supporting_citations":[{"cited_title":"Benchmarking neu- ral network robustness to common corruptions and perturba- tions","cited_arxiv_id":null,"evidence_quote":"Defines the 15 2D Common Corruptions that form the synthetic corruption test set."},{"cited_title":"ACDC: The adverse conditions dataset with correspondences for se- mantic driving scene understanding","cited_arxiv_id":null,"evidence_quote":"Provides the ACDC real-world adverse-conditions dataset used as the real-world reference."},{"cited_title":"The cityscapes dataset for semantic urban scene understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the Cityscapes dataset on which models are trained and on which synthetic corruptions are applied."},{"cited_title":"Benchmarking the robustness of semantic segmentation models","cited_arxiv_id":null,"evidence_quote":"Prior robustness benchmark for semantic segmentation that this study extends to more architectures and to real-world correlations."}],"review_version":1}