{"id":"90aaadd5-8982-46a1-b8b7-55ce0732a2fb","arxiv_id":"2505.05091","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"DispBench measures four stereo disparity models under 15 common corruptions and 5 adversarial attacks, finding transformer-based models more fragile on weather corruptions and no reliable transfer from synthetic to real corruption benchmarks.","lead":"This paper introduces DispBench, an open benchmarking tool that measures how stereo disparity estimation models behave under synthetic image corruptions and adversarial attacks. It reports that newer transformer-based models are often less robust than older CNNs, and that corrupting synthetic data does not predict real-world robustness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Section 5 proxy-failure claim is confounded: Figure 6 compares different checkpoints trained on different datasets, so the observed non-correlation can reflect training-domain differences, not synthetic-versus-real corruption; with only two models there is no statistical basis for the…","rationale":"The reader's weakest_assumption identifies the same load-bearing flaw: comparing FT3D-trained checkpoints evaluated on corrupted FT3D with KITTI-trained checkpoints evaluated on corrupted KITTI confounds the training distribution with the dataset domain. My read confirms this is the central, load-bearing problem. The claim in Section 5 and the Conclusion is far stronger than the evidence: it asserts that synthetic corruptions on synthetic datasets do not represent real-world corruptions, and even concludes that synthetic simulators cannot be used for real-world applications. That conclusion would require a controlled comparison where only the corruption target domain changes while model weights are held constant, and enough architectures and repeated seeds to establish a correlation or its absence. The paper provides neither: at most two KITTI checkpoints are available, no correlation statistics are reported, and no matched-scene control exists. The secondary transformer-fragility claim is also underpowered, but the primary reason for rejecting the paper as written is the unsupported proxy-failure claim. The benchmark artifact itself could be useful if the claims were removed or reframed, but the submitted manuscript's central scientific contribution does not hold. I therefore concur with the reader's REJECT verdict, with no change needed.","tokens_in":22047,"tokens_out":3947,"duration_ms":40790,"concrete_test":"Hold the checkpoint fixed: take the FT3D-pretrained STTR and GWCNet-G checkpoints, evaluate them zero-shot on corrupted KITTI2015 validation images and on corrupted FT3D validation images, and compute the Spearman rank correlation of the 15 per-corruption EPE values at each severity. If a positive correlation emerges (e.g., > 0.5), the Figure 6 'no correlation' conclusion is an artifact of the training-dataset/checkpoint mismatch. As a second check, fine-tune both architectures on both datasets from the same initialization with five random seeds and recompute the correlation using seed-averaged errors; if the correlation is not robust across seeds, the claim lacks statistical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'synthetic corruptions on synthetic datasets do not represent real-world corruptions' rests on Figure 6, which correlates EPE on corrupted FlyingThings3D using an FT3D-trained checkpoint with EPE on corrupted KITTI2015 using a KITTI-trained checkpoint. This varies two factors at once: the corruption target dataset (synthetic versus real) and the model weights (trained on FT3D versus trained on KITTI). The observed lack of correlation between the two measurements cannot therefore be attributed to the synthetic-versus-real corruption gap; it may simply reflect that robustness of an FT3D-trained model to corruptions of its training domain is not predictive of robustness of a KITTI-trained model to corruptions of its own domain. The paper itself states in Section 4.2 that only GWCNet-G and STTR have KITTI2015 checkpoints, so Figure 6 contains at most two meaningful points per severity; with two points, any correlation coefficient is either +1 or -1 and no 'no correlation' conclusion is statistically supported. The transitivity argument also imports a semantic-segmentation result from Agnihotri et al. [7] (Cityscapes-to-ACDC) and assumes without evidence that the same proxy relation holds for disparity estimation. Even if the checkpoint confound were removed, the two datasets differ in scene content, disparity range, and resolution, so a further matched-scene or domain-controlled comparison would be needed before concluding that synthetic simulators such as CARLA, LGSVL, and AirSim are unusable for robustness evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DispBench, an open-source benchmarking tool for evaluating stereo disparity estimation methods under synthetic 2D common corruptions and white-box adversarial attacks. The authors benchmark four architectures (CFNet, GWCNet-G, STTR, STTR-light) on FlyingThings3D and, where checkpoints exist, on KITTI2015, reporting mean end-point error under fifteen common corruptions at five severity levels. The main empirical findings are that transformer-based methods are less robust than CNN-based methods to weather corruptions on FlyingThings3D, and that the relative robustness ordering is not preserved on KITTI2015. Based on a scatter-plot correlation between FlyingThings3D and KITTI2015 evaluations, Section 5 concludes that synthetic corruptions on synthetic datasets do not represent real-world corruptions, and the conclusion then infers that synthetic simulators such as CARLA, LGSVL, and AirSim are unsuitable for real-world applications.","tokens_in":22296,"tokens_out":4775,"duration_ms":43189,"significance":"The paper's strongest contribution is a reproducible evaluation harness with released code and logs, enabling standardized robustness testing for disparity estimation; the measurements themselves appear carefully computed with standard metrics. If the proxy-failure claim were established, it would have substantial implications for the use of synthetic simulators and synthetic corruptions in stereo robustness research. However, the central claim rests on a comparison that confounds training distribution with corruption target domain and on a two-point correlation per severity level, so the evidence as presented does not support the headline conclusion. The finding that transformer-based STTR variants degrade more than older CNN methods under weather corruptions on FlyingThings3D is interesting and better supported, though it too would benefit from additional architectures and repeated seeds.","major_comments":[{"comment":"The claim that synthetic corruptions on synthetic datasets do not represent synthetic corruptions on real-world datasets is based on correlating EPE on corrupted FlyingThings3D obtained with FlyingThings3D-pretrained checkpoints against EPE on corrupted KITTI2015 obtained with KITTI2015-pretrained checkpoints. This varies two factors at once (the corruption target domain and the training distribution), so the observed lack of correlation cannot be uniquely attributed to the synthetic-versus-real nature of the corruption; the effect may instead reflect the different training domains. A controlled comparison, such as evaluating the same checkpoint on both datasets or training identical architectures on both datasets, is needed.","section":"Section 5, Figure 6"},{"comment":"Because only GWCNet-G and STTR have public KITTI2015 checkpoints, each severity panel in Figure 6 contains only two points, and each per-corruption panel in Figure 9 contains only two points as well. With two points, a Pearson correlation coefficient is necessarily either +1 or -1, so the figures cannot support the conclusion that there is 'no correlation in performance.' Reporting a correlation statistic over the full set of corruptions, or benchmarking additional architectures with KITTI2015 checkpoints, would be required to support the proxy-failure claim.","section":"Section 4.2, Figure 6, Figure 9"},{"comment":"The transitivity argument that 'synthetic corruptions on real-world datasets represent real-world corruptions' is imported from Agnihotri et al. [7], which demonstrates a correlation between Cityscapes Common Corruptions and ACDC for semantic segmentation. No evidence is provided that this proxy relation transfers to disparity estimation; the cited work is by the same authors and concerns a different task. Since this assumption is load-bearing for the central claim, the conclusion in Section 5 is not established by the experiments in this manuscript.","section":"Appendix A, Section 5"},{"comment":"The conclusion that the findings 'eliminate the possibility of using synthetic simulators like CARLA, LGSVL, AirSim' overreaches the evidence: the experiments only compare 2D corruptions applied to FlyingThings3D and KITTI2015 images, not the use of simulators for generating training data or novel scenes. Even if the proxy-failure claim were supported, the conclusion should be limited to the use of 2D Common Corruptions on synthetic datasets as proxies for real-world corruptions.","section":"Section 6, Conclusion"}],"minor_comments":[{"comment":"Appendix B says 'we focus on generating 3D Common Corruptions for now,' but no 3D corruption results appear anywhere in the paper, and the abstract and contributions only mention 2D Common Corruptions; this inconsistency should be resolved.","section":"Appendix B"},{"comment":"Appendix C lists MPI Sintel as a supported dataset, while Section 2.4 and the contribution list state that two datasets are supported; these counts should be harmonized.","section":"Appendix C and Section 2.4"},{"comment":"There are typos in the article and figures, including 'Lipshitz' for 'Lipschitz,' 'Syntheticc Corrruptions' in Figure 1, 'GWCNet-C' in the Figure 5 caption, and 'dispcbench' in the code example in Section 2.2; these should be corrected.","section":"Throughout"},{"comment":"The code example in Section 2.3 for 2D Common Corruptions appears to be a copy of the adversarial-attack example with a different configuration; it should be checked so that the shown call matches the intended threat model.","section":"Section 2.3"}],"recommendation":"reject","confidential_remarks":"The manuscript reuses substantial text from the authors' prior work on FlowBench [6] and [7], and the appendix contains many duplicated passages; while this is not grounds for rejection by itself, the editor may wish to consider whether the novelty relative to those prior works is sufficient for a journal publication. In addition, the central proxy-failure claim relies on the authors' own prior semantic-segmentation result, making the evidence less independent than it appears."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key takeaways: DispBench is a useful benchmark artifact, and the per-corruption measurements are probably fine. The paper's load-bearing scientific claim — that synthetic corruptions on synthetic datasets don't represent real-world corruptions — is not supported by the data it presents.\n\nWhat's genuinely new: it's the first standardized robustness benchmark for disparity estimation that applies the RobustBench/FlowBench recipe (15 2D Common Corruptions at 5 severities, plus 5 attacks) to four models on two datasets. Figures 4 and 5 are exactly the kind of per-corruption curves the field needs, and the codebase looks designed to be extended. The observation that STTR/STTR-light collapse more than CNN-based models under snow/frost/fog on FlyingThings3D is worth reporting, even if it's preliminary.\n\nWhere it falls down:\n\n1. Section 5 and Figure 6. The proxy-failure claim compares an FT3D-trained checkpoint evaluated on corrupted FT3D with a KITTI-trained checkpoint evaluated on corrupted KITTI. That varies two factors at once — training distribution and evaluation domain — so the lack of correlation cannot be pinned on the synthetic-vs-real corruption gap. Worse, only two KITTI checkpoints exist (GWCNet-G, STTR), so each panel has at most two points; you can't draw a 'no correlation' conclusion from that. The transitivity from [7] — a semantic-segmentation result on Cityscapes/ACDC — is asserted, not shown, for disparity estimation. The 'CARLA/LGSVL/AirSim are unusable' takeaway is far beyond the evidence.\n\n2. The manuscript is internally sloppy. Appendix B says the benchmark supports 3 datasets (including MPI Sintel) and says they're focusing on 3D Common Corruptions; the main text says 2 datasets and only 2D CC. Appendix A is largely copy-pasted from the optical flow paper, including a sentence about 'this work' evaluating 3D CC and a remark that no in-the-wild dataset exists for optical flow. Appendix C mentions attacks (PCFA, Adversarial Weather) that aren't in the supported list. These are fixable, but they make the paper read like a draft.\n\n3. The reproducibility statement promises logs and checkpoints but gives no commit hash or artifact ID; the code URL is a branch. That's not independently verifiable from the paper.\n\nThe benchmark measurements themselves are standard and likely correctly computed. If the Section 5 claim were removed, or re-run with a clean design (same checkpoint evaluated on both datasets, or more architectures with correlation statistics), the benchmark part would be a solid contribution.\n\nWho should read it: anyone building or evaluating stereo methods who needs a starting point for robustness testing. It deserves a serious referee — I'd send it to review rather than desk-reject — but with the expectation that the central claim is either cut or reworked.","headline":"Useful benchmark artifact, but the paper's central claim about synthetic-to-real proxies is built on a confounded comparison and doesn't survive scrutiny.","tokens_in":22882,"tokens_out":4898,"would_cite":false,"duration_ms":45402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that synthetic corruptions applied to synthetic datasets do not predict how disparity-estimation models behave under the same corruptions on real data, and that driving simulators therefore cannot stand in for real-world…","keywords":["disparity estimation","stereo matching","robustness benchmark","2D Common Corruptions","adversarial attacks","out-of-distribution generalization","transformer robustness","synthetic-to-real gap"],"falsifier":"Train several stereo-matching architectures on the same synthetic data, evaluate them on corrupted synthetic data, then fine-tune the same architectures on real data and evaluate them on corrupted real data; if their robustness rankings correlate strongly across a larger pool of models, the no-proxy conclusion would be refuted.","tokens_in":21768,"feed_emoji":"🌫️","tokens_out":4775,"duration_ms":48150,"temperature":0.7,"pith_summary":"This paper introduces DispBench, a benchmarking tool for testing disparity-estimation models against synthetic image corruptions and adversarial attacks across two datasets. Using it, the authors report that older CNN-based stereo models and newer transformer-based models all degrade sharply under noise and weather corruptions, with the transformer models degrading most under snow, frost, and fog on the synthetic dataset. The central, load-bearing claim is that corruption robustness measured on a synthetic dataset does not transfer to the same corruptions applied to a real dataset, so synthetic-only robustness scores mislead real-world safety judgments. A sympathetic reader should care because disparity estimation is used in medical surgery and autonomous navigation, where reliability under realistic image degradation matters as much as clean benchmark accuracy.","feed_headline":"Synthetic corruption tests mislead real-world stereo model checks","feed_subtitle":"Transformer stereo models are most fragile under weather corruptions, so corrupted real-world data is needed.","key_machinery":"The load-bearing mechanism is DispBench, a benchmark harness wrapping four disparity-estimation architectures, two datasets (FlyingThings3D and KITTI2015), five white-box adversarial attacks (FGSM, BIM, PGD, APGD, and CosPGD), and fifteen 2D Common Corruptions at five severities, reported through mean End-Point Error. The decisive analytical move is the correlation comparison: for each architecture, robustness on corrupted synthetic data is plotted against robustness on corrupted real data at each severity, and the absence of correlation is what carries the no-proxy conclusion.","core_discovery":"The paper's central discovery is that synthetic corruptions on synthetic datasets do not represent synthetic corruptions on real-world datasets, and by extension do not represent real-world corruptions. The evidence is a per-architecture comparison: for the same stereo-matching architecture, mean End-Point Error on corrupted FlyingThings3D is plotted against mean End-Point Error on corrupted KITTI2015 at several corruption severities, and no correlation appears. The authors conclude that performance on corrupted synthetic images cannot be used as a proxy for performance on corrupted real images, and therefore synthetic driving simulators cannot be used to validate disparity-estimation methods for real-world deployment. A second finding is that the newer transformer-based methods STTR and STTR-light are significantly more non-robust than older CNN-based methods under weather corruptions such as snow, frost, and fog on FlyingThings3D.","pith_inferences":["A natural extension is that the no-proxy result would likely also apply to other geometric vision tasks such as optical flow, since the mechanism is about the gap between synthetic and real image distributions rather than about stereo matching specifically.","A stronger test of the paper's claim would compute rank correlations across a larger pool of models, training seeds, and checkpoints; the current comparison has only two or four architectures per severity, so the observed lack of correlation may partly reflect training-distribution differences.","If the synthetic-to-real corruption gap is the real driver, then domain-randomized simulators that add realistic sensor noise and weather effects could partially restore predictivity, offering a testable path to cheaper real-world validation."],"forward_implications":["If the no-proxy claim is correct, robustness results reported only on corrupted synthetic datasets should not be read as evidence of real-world reliability.","Synthetic driving simulators cannot substitute for corrupted real-world data when validating disparity-estimation models for safety-critical uses.","Newer transformer-based disparity-estimation methods need explicit attention to weather-corruption robustness before deployment.","Benchmarking effort should shift toward collecting or generating corrupted real-world stereo pairs with ground truth, since synthetic corruption alone cannot certify generalization."],"supporting_citations":[{"why":"Supplies the prior finding that synthetic corruptions on real-world images correlate strongly with real-world domain shifts, which the paper contrasts with its synthetic-dataset result.","marker":"[7]"},{"why":"Defines the fifteen 2D Common Corruptions and five severities that form the corruption suite used throughout the benchmark.","marker":"[31]"},{"why":"Provides the FlyingThings3D synthetic dataset used as one of the two evaluation domains and as the training source for the synthetic checkpoints.","marker":"[41]"},{"why":"Provides the KITTI2015 real-world driving dataset used as the second evaluation domain and as the training source for the real-world checkpoints.","marker":"[44]"},{"why":"Supplies the STTR and STTR-light transformer architectures whose weather-corruption fragility is a key finding.","marker":"[39]"},{"why":"Supplies the GWCNet-G architecture, the older CNN-based method used in the synthetic-versus-real correlation analysis.","marker":"[26]"},{"why":"Supplies the CFNet architecture, the second older CNN-based method used in the robustness evaluations.","marker":"[55]"},{"why":"Establishes the threat-model evaluation paradigm that DispBench adopts for structuring its robustness evaluations.","marker":"[12]"},{"why":"Represents the synthetic driving simulators whose real-world validation use the paper argues against based on its no-proxy finding.","marker":"[18]"}],"fun_headline_variants":["Synthetic corruptions misrepresent real-world stereo failures","Stereo models fail differently on real vs synthetic corruptions","Transformer stereo models weakest under weather corruptions","No correlation: synthetic corruptions don't predict real stereo errors","DispBench: synthetic corruptions can't validate stereo models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that synthetic corruptions do not predict real-world corruptions compares models trained on different datasets, so the synthetic and real results differ in both the corruption target and the training data; the lack of correlation could therefore reflect training-domain differences rather than a synthetic-versus-real corruption gap.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic corruptions misrepresent real-world stereo failures","Stereo models fail differently on real vs synthetic corruptions","Transformer stereo models weakest under weather corruptions","No correlation: synthetic corruptions don't predict real stereo errors","DispBench: synthetic corruptions can't validate stereo models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2243,"prompt_tokens":922,"completion_tokens":1321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":1242}},"tokens_in":538,"tokens_out":1321,"duration_ms":8820,"temperature":1.0,"reasoning_tokens":1242,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:13:07.900025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train several stereo-matching architectures on the same synthetic data, evaluate them on corrupted synthetic data, then fine-tune the same architectures on real data and evaluate them on corrupted real data; if their robustness rankings correlate strongly across a larger pool of models, the no-proxy conclusion would be refuted.","supporting_citations":[{"cited_title":"Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions? In CVPR Workshop On Synthetic Data for Computer Vision,","cited_arxiv_id":null,"evidence_quote":"Supplies the prior finding that synthetic corruptions on real-world images correlate strongly with real-world domain shifts, which the paper contrasts with its synthetic-dataset result."},{"cited_title":"Benchmarking neu- ral network robustness to common corruptions and perturba- tions","cited_arxiv_id":null,"evidence_quote":"Defines the fifteen 2D Common Corruptions and five severities that form the corruption suite used throughout the benchmark."},{"cited_title":"A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation","cited_arxiv_id":null,"evidence_quote":"Provides the FlyingThings3D synthetic dataset used as one of the two evaluation domains and as the training source for the synthetic checkpoints."},{"cited_title":"Object scene flow for au- tonomous vehicles","cited_arxiv_id":null,"evidence_quote":"Provides the KITTI2015 real-world driving dataset used as the second evaluation domain and as the training source for the real-world checkpoints."},{"cited_title":"Creighton, Russell H","cited_arxiv_id":null,"evidence_quote":"Supplies the STTR and STTR-light transformer architectures whose weather-corruption fragility is a key finding."},{"cited_title":"Group-wise correlation stereo network","cited_arxiv_id":null,"evidence_quote":"Supplies the GWCNet-G architecture, the older CNN-based method used in the synthetic-versus-real correlation analysis."},{"cited_title":"Cfnet: Cascade and fused cost volume for robust stereo matching","cited_arxiv_id":null,"evidence_quote":"Supplies the CFNet architecture, the second older CNN-based method used in the robustness evaluations."},{"cited_title":"RobustBench: a standardized adversarial robustness benchmark","cited_arxiv_id":null,"evidence_quote":"Establishes the threat-model evaluation paradigm that DispBench adopts for structuring its robustness evaluations."},{"cited_title":"CARLA: An open urban driving simulator","cited_arxiv_id":null,"evidence_quote":"Represents the synthetic driving simulators whose real-world validation use the paper argues against based on its no-proxy finding."}],"review_version":1}