{"id":"4f449cbe-ce07-47bb-8f2f-995d19fca727","arxiv_id":"2504.18510","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Models lose far more accuracy to realistic optical aberrations than to the simple disk blur used in standard benchmarks, and training on simulated lens blur partially closes the gap.","lead":"This paper builds new image-blur benchmarks from real camera lens designs and shows that common simple blur tests understate how much optical aberrations hurt vision models. A companion training method uses the same lens kernels to make models more robust.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OpticsBench corruptions confound kernel shape with chromatic aberration, so the weak rank correlations do not isolate the 'shape' explanation.","rationale":"I read the paper in good faith: the strongest claim is not merely that optical blur differs from disk blur, but that the shape of the kernel is what makes disk blur a weak proxy. The extensive evaluation and rank-correlation analysis are appropriate for the broad claim. The load-bearing condition is that the comparison isolates the shape variable. It does not. Supplementary E-A explicitly initializes the wavefront with color-dependent Zernike coefficients, so OpticsBench corruptions carry chromatic aberration; the Hendrycks defocus baseline is a grayscale disk. Thus the observed tau values are evidence against the disk as a full optical-degradation proxy, not specifically against the disk shape. This is a confound internal to the benchmark, independent of the realism of nominal lens prescriptions. The reader's realism concern matters for LensCorruptions as a public benchmark, but would not resolve this confound. I would not reject the paper: the datasets and evaluation are still valuable, and the broad recommendation to include optical kernels is probably correct. However, the paper should ablate color before attributing the result to shape. A monochromatic OpticsBench rerun is cheap and decisive; if the code is released (the current URL is malformed), this should be straightforward. Hence the verdict remains conditional, not changed.","tokens_in":51163,"tokens_out":7011,"duration_ms":78942,"concrete_test":"Generate monochromatic 'gray-shape' variants of the OpticsBench kernels at severity 4 by replacing the R, G, B channels with a single luminance PSF (e.g., the green-channel PSF or the energy-summed PSF), L1-normalized and re-matched to the defocus baseline using the paper's own MTF/AUC criteria. Evaluate the same 72 ImageNet-1k models on these gray-shape corruptions and recompute Kendall tau against the Hendrycks defocus baseline. If tau rises from roughly 0.2-0.4 to above about 0.7, the weak-proxy result is driven by chromatic aberration rather than kernel shape; if tau stays below about 0.4, the shape conclusion is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that disk-shaped blur kernels are weak proxies because they ignore the shape of optical PSFs (Abstract, Introduction). To support this, the paper compares model rankings on the Hendrycks defocus baseline with rankings on OpticsBench and finds Kendall tau around 0.2-0.4 (Sec. VI-A, Fig. 5). But the OpticsBench kernels are not shape-matched to the baseline: Eq. 5 generates a per-wavelength PSF from a wavefront that includes a chromatic baseline (Suppl. E-A, Table XVII: different Zernike coefficients for R, G, B), so each OpticsBench corruption is an RGB kernel with color fringes, whereas the disk baseline is a single grayscale kernel applied to all channels. The design therefore varies two factors at once: kernel geometry and chromatic/color-channel structure. The weak rank correlation, and the coma non-significance, could be driven entirely by chromatic aberration rather than by the non-circular PSF shape. The paper's mechanistic conclusion ('different blur kernel shapes', Abstract) is not identified by this experiment. LensCorruptions has the same confound because its Huygens PSFs are computed at F, d, C wavelengths and stacked into RGB kernels (Suppl. D-A). Without an ablation that removes color, the headline comparison cannot tell the community whether it needs realistic kernel shapes or just chromatic variation.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two new blur-corruption benchmarks for vision models: OpticsBench, built from Zernike-mode PSFs for primary aberrations (astigmatism, coma, defocus with spherical, trefoil), and LensCorruptions, built from Huygens-traced PSFs of 100 curated real lens prescriptions. The authors evaluate 72 ImageNet classifiers, 8 MSCOCO detectors (plus NuImages), report TIDE error decompositions, and compare model rankings on their corruptions against the disk-shaped defocus baseline of Hendrycks et al. They report weak Kendall rank correlations and argue that disk-shaped kernels are weak proxies for optical blur. They also propose an augmentation method, OpticsAugment, trained on OpticsBench kernels, and report gains on OpticsBench and 2D common corruptions. The central claims are that realistic optical kernel shapes matter for robustness evaluation and that optical kernels are a useful training augmentation.","tokens_in":51290,"tokens_out":6405,"duration_ms":66564,"significance":"If the central claim holds, the paper makes a useful benchmark contribution: it provides two reusable corruption suites, a large lens-derived kernel database, a validation of a 1000-image COCO subset against the full set (0.2% mAP difference), and an unusually broad evaluation across architectures, including TIDE error decompositions. The LensCorruptions curation from 718 lenses and the explicit code release are also valuable. However, the load-bearing comparison that supports the 'shape matters' conclusion is confounded: OpticsBench kernels differ from the disk baseline both in PSF geometry and in chromatic structure, so the weak rank correlations do not identify shape as the cause. The realism of LensCorruptions also rests on an untested pixel-size and nominal-lens assumption. These issues are fixable with additional analyses, but they are central to the paper's message, so the manuscript needs revision before the claims are supported.","major_comments":[{"comment":"The central claim that disk-shaped blur kernels are weak proxies because of 'different blur kernel shapes' is not identified by the reported experiment. The OpticsBench corruptions are generated by Eq. 5 per wavelength with a chromatic baseline (Suppl. E-A, Table XVII gives different Zernike coefficients for R, G, B), so each OpticsBench kernel is an RGB kernel with color fringes, while the Hendrycks baseline applies the same grayscale disk to all channels. The comparison therefore varies two factors at once: kernel geometry and chromatic/color-channel structure. The weak Kendall tau values (around 0.2–0.4, with coma non-significant at p=0.5) could be driven entirely by chromatic aberration. Please add an ablation that removes the color structure, e.g. grayscale versions of the OpticsBench kernels or disk kernels carrying the same per-channel chromatic variation, so that the shape hypothesis is tested in isolation.","section":"§VI-A, Fig. 5, Table III; Suppl. E-A, Table XVII"},{"comment":"The realism of LensCorruptions rests on two untested assumptions: setting the virtual pixel size by matching the sensor Nyquist frequency to the mean MTF20 value, and treating nominal lens prescriptions as representative of real lenses on the grounds that manufacturing tolerances are the only difference. If the MTF20 criterion is not appropriate, or if real lenses deviate substantially from their nominal designs, the severity matching and the 'realistic' qualification of LensCorruptions are weakened. Please add a sensitivity analysis, for example recomputing a subset of kernels under a different pixel-size criterion (MTF50, or a 2×MTF20 Nyquist choice) and comparing selected kernels against the toleranced PSF shown in the supplementary, to bound the effect on the reported accuracies and mAP values.","section":"§III-B and §V-A"},{"comment":"The OpticsAugment evaluation is partly circular: the augmentation is trained with kernels drawn from the same OpticsBench family on which it is evaluated, so the headline gains (about 18% average improvement on ImageNet-100 OpticsBench, and +7.7% mAP on MSCOCO-OpticsBench) are by construction to some degree. The transfer results to 2D common corruptions are encouraging and should be kept as independent evidence, but the paper does not evaluate OpticsAugment on LensCorruptions, which are the unseen realistic kernels. Please report the transfer to LensCorruptions, or an ablation that excludes the exact training kernels and severities, before concluding that the specific blur kernels can be compensated 'to a large extent.'","section":"§VII-A, Tables VI and VIII"},{"comment":"The manuscript sometimes presents the disk baseline as uniquely weak, but the supplementary rank-correlation matrix shows that the OpticsBench corruptions are also only weakly correlated with each other (e.g. tau around 0.16–0.32 among astigmatism, coma, trefoil, defocus&spherical). If every single blur corruption is a weak proxy for others, the conclusion that a particular shaped kernel is needed should be phrased more carefully. Please either compare the disk baseline against another non-optical blur corruption (e.g. Gaussian blur) with the same analysis, or explicitly state that all single-kernel proxies are weak and the contribution is the availability of more diverse optical kernels.","section":"§VI-A and Suppl. F-A3, Fig. 25"}],"minor_comments":[{"comment":"The code URL contains a space ('classification robustness'); it should be URL-encoded or replaced with the repository's canonical URL.","section":"Code footnote"},{"comment":"The signed improvements in Table XXV appear to use the opposite sign convention from Table IX: for many corruptions the cascade column is numerically higher than the OpticsAugment-only column while the delta column is negative, whereas Table IX reports positive 'additional average improvement.' Please define the delta explicitly and make the signs consistent between the main text and supplement.","section":"Suppl. Table XXV vs Table IX"},{"comment":"The sentence 'Besides depth, the simulation does not cover lens flare or scattering effects' is awkward; the first phrase should be clarified, for example by stating explicitly that depth-dependence is dropped for fixed-focus lenses and that other imaging effects are out of scope.","section":"§V-A"},{"comment":"The ImageNet-100 subset is attributed to reference [79] in a way that requires the reader to know the exact split from the Contrastive Multiview Coding paper; consider citing the specific ImageNet-100 split used or describing the selection procedure in one sentence.","section":"§VIII-A"},{"comment":"The simulator console output embedded in Fig. 22 is visually noisy and partly redundant with the plotted MTF curves; consider cropping the console text or moving it to the caption/appendix for readability.","section":"Fig. 22"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial benchmark contribution with broad evaluation and released code, but the mechanistic claim about kernel shape needs an ablation that removes the chromatic confound. The LensCorruptions realism assumption also needs a sensitivity analysis. These are within scope of a major revision rather than grounds for rejection. I would ask the authors to add the ablation and the LensCorruptions transfer result for OpticsAugment before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is LensCorruptions: 100 blur corruptions sampled from 718 real lens prescriptions, with careful curation by MTF quality, field positions, and a sensible MTF20-based virtual pixel size. The evaluation scale is real—72 classification and 8 detection models, TIDE error decomposition, rank correlations with p-values, and a validated MSCOCO subset. That alone is worth having, and the object-detection extension over the ICCV workshop paper is a genuine addition.\n\nThe central claim, though, is oversold. The paper says disk-shaped blur kernels are weak proxies because they ignore the shapes of optical PSFs. The evidence is the low Kendall tau between model rankings on the grayscale disk baseline and on OpticsBench. But OpticsBench kernels are RGB—they carry a chromatic baseline with different Zernike coefficients per color channel, plus color fringes. The disk baseline is a single grayscale kernel. So the comparison varies kernel geometry and chromatic structure at the same time. The weak rank correlation could be driven mostly by color. The paper even shows the fringes in Figure 1. Without an ablation that removes color (e.g., luminance-only optical kernels), the mechanistic \"shape\" conclusion is not identified. The broader practical point—a simple grayscale disk is a poor proxy for realistic optical degradation—probably survives, because real lenses are chromatic. But the explanation in the abstract is not what the experiment shows.\n\nThe OpticsAugment result has a known circularity burden: kernels from the same family are used in training, so the large gains on OpticsBench are partly by construction. Transfer to ImageNet-C helps, but the paper should separate in-distribution from transfer results clearly. Also, the abstract's 18% average gain doesn't match Table VI, which averages closer to 15.5 points. Minor but sloppy. Reproducibility is weakened by a malformed code URL and no dataset release or commit hash.\n\nThe imaging model's simplifications—no sensor noise, no ISP, no depth—are stated honestly and are reasonable for a blur-focused study. The MTF20 pixel-size rule is a free parameter, but the argument for it is at least explicit.\n\nThis paper is for people building robustness benchmarks or linking camera quality to model performance. It deserves a serious referee, but needs major revision to decouple shape from color and to fix the reporting issues.\n\nRecommendation: send to peer review, with the confound and reproducibility raised as must-fix.","headline":"A genuinely useful optical-blur robustness benchmark, but the headline claim that kernel shape drives the weak correlation with disk blur is not isolated from chromatic aberration.","tokens_in":51941,"tokens_out":2536,"would_cite":true,"duration_ms":24316,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real optical blur re-ranks vision models differently than the standard defocus-blur test, two new benchmarks show.","keywords":["optical aberrations","robustness benchmark","Zernike polynomials","point spread function","image classification","object detection","data augmentation","lens blur"],"falsifier":"Take a set of real cameras with measured PSFs (for example from a lens bench), generate the same corruptions with those measured PSFs, and rank a dozen vision models on them; if the resulting ranking matches the ranking on the disk-defocus baseline with high Kendall tau, the paper's central claim about disk kernels being weak proxies would be contradicted. A cheaper check: replace the MTF20 rule with MTF50 or a fixed pixel pitch and test whether the LensCorruptions rankings change materially.","tokens_in":50829,"feed_emoji":"📷","tokens_out":3648,"duration_ms":32524,"temperature":0.7,"pith_summary":"This paper argues that the standard way of testing vision models against blur—convolving with simple disk-shaped defocus kernels—does not rank models the way realistic optical blur does. It introduces two benchmark families: OpticsBench, which varies single Zernike aberrations (coma, astigmatism, defocus-and-spherical, trefoil), and LensCorruptions, which uses ray-traced point spread functions from 100 real lens prescriptions. Across ImageNet classification and MSCOCO/NuImages detection, model performance varies with the aberration type, and the rank correlation between the disk-blur baseline and the optical corruptions is weak (Kendall tau around 0.2–0.4, with coma not statistically significant). The paper also shows that training with an aberration-aware augmentation (OpticsAugment) closes much of the gap on OpticsBench and transfers some robustness to standard 2D corruptions. A fair reader should take away that realistic blur kernels belong in robustness evaluation.","feed_headline":"Simple defocus blur mis-ranks vision models under real lens blur","feed_subtitle":"Two new corruption datasets show disk-shaped kernels are weak proxies for optical aberrations; a targeted augmentation closes much of the…","key_machinery":"The engine is the point spread function (PSF), derived from the wavefront aberration $W_\\lambda$ expanded into Zernike polynomials for single aberrations, or obtained by tracing rays through a lens prescription following the Huygens principle for real lenses. A critical modeling step matches each lens's PSF to a virtual pixel size using an MTF20 criterion: the sensor's Nyquist frequency is set so the lens retains 20% contrast at that frequency, fixing the otherwise free pixel-size parameter and yielding pixel sizes between 1 and 20 $\\mu$m. The PSFs are then convolved with images; for LensCorruptions each lens supplies five field positions and three azimuth orientations, giving 500 corrupted datasets.","core_discovery":"The central discovery is that the shape of the blur kernel matters for model robustness, not just its size. Using Zernike polynomials to encode wavefront aberrations and Huygens ray tracing on real lens prescriptions, the authors generate point spread functions for coma, astigmatism, defocus-and-spherical, trefoil, and 100 real lenses. Across more than 65 classifiers and 8 detectors, they find that a model's rank on the disk-defocus baseline from Hendrycks et al. is only weakly predictive of its rank on these optical corruptions: Kendall tau values cluster around 0.2–0.4, and the coma ranking does not correlate significantly with the baseline (p = 0.5). The authors conclude that the disk-shaped kernel is a weak proxy for realistic optical degradation and that optical kernels should be part of robustness evaluation.","pith_inferences":["If the weak rank correlation holds across a wider model zoo, then any robustness evaluation that reports a single 'blur' number is likely averaging over incompatible failure modes; splitting blur by aberration type would give model developers more actionable information.","The MTF20-based pixel-size rule implies a testable prediction: real cameras whose pixel pitch deviates from MTF20 matching should show even larger discrepancies from OpticsBench severities than the simulated ones.","LensCorruptions could be extended into a training-time regularizer by sampling lens PSFs directly, potentially covering the gap between simple disk blur and full optical models without the need for a separate augmentation step."],"forward_implications":["Robustness rankings from defocus-blur benchmarks do not transfer to optical aberrations, so benchmarks that use only disk-shaped kernels can mislead model selection for cameras in safety-critical systems.","Models trained with OpticsAugment gain on average 18% accuracy on OpticsBench (ImageNet-100) and improve robustness to standard 2D common corruptions by about 5.3 points, suggesting that aberration-aware augmentation is a practical remedy.","Lens quality as measured by MTF50 correlates strongly with model accuracy (Pearson $\\rho \\approx 0.86$ for classification and $\\rho \\approx 0.89$ for detection), meaning optical design quality can directly predict vision performance.","Accuracy and mAP drop with distance from the lens center, so field-dependent optical effects are measurable and should be included in camera-system evaluation."],"supporting_citations":[{"why":"Supplies the baseline disk-shaped defocus blur and the common-corruptions evaluation protocol that OpticsBench kernels are matched against and compared with.","marker":"[7]"},{"why":"The predecessor workshop paper that introduces the primary-aberration kernels and the OpticsAugment method here extended and consolidated.","marker":"[11]"},{"why":"Provides the wavefront aberration and Zernike polynomial formalism underlying the PSF derivation from the exit pupil.","marker":"[1]"},{"why":"The ray-tracing software used to compute Huygens PSFs from the lens prescriptions in LensCorruptions.","marker":"[12]"},{"why":"Gives the Fourier-optics relation between the pupil function and the PSF used to generate kernels from the wavefront.","marker":"[45]"},{"why":"The public lens-design database from which the 718 (later 100) nominal lens prescriptions were drawn and curated.","marker":"[52]"},{"why":"The RobustBench leaderboard supplying the adversarial- and corruption-robust ResNet50 checkpoints that are compared with the evaluated models.","marker":"[21]"},{"why":"AugMix, the augmentation baseline that OpticsAugment is compared against and cascaded with for robustness transfer.","marker":"[26]"}],"fun_headline_variants":["Disk blur misranks vision models; real optics change robustness","Real lens aberrations beat disk blur in robustness tests","OpticsBench: Why simple defocus fails to measure blur robustness","Blur kernel shape alters model robustness rankings","Zernike-based blur datasets reveal disk kernel's blind spot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes that PSFs computed from nominal lens prescriptions, with the virtual pixel size fixed by the MTF20 criterion, faithfully represent what a real camera of that lens would produce; if real lenses deviate from their nominal designs beyond manufacturing tolerances, or if the MTF20 rule does not reflect how sensors are actually matched, then the LensCorruptions severities and the comparison to disk blur are off.","fun_headline_variants_meta":{"raw":{"variants":["Disk blur misranks vision models; real optics change robustness","Real lens aberrations beat disk blur in robustness tests","OpticsBench: Why simple defocus fails to measure blur robustness","Blur kernel shape alters model robustness rankings","Zernike-based blur datasets reveal disk kernel's blind spot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1484,"prompt_tokens":945,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":458}},"tokens_in":561,"tokens_out":539,"duration_ms":5126,"temperature":1.0,"reasoning_tokens":458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:14:53.685677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of real cameras with measured PSFs (for example from a lens bench), generate the same corruptions with those measured PSFs, and rank a dozen vision models on them; if the resulting ranking matches the ranking on the disk-defocus baseline with high Kendall tau, the paper's central claim about disk kernels being weak proxies would be contradicted. A cheaper check: replace the MTF20 rule with MTF50 or a fixed pixel pitch and test whether the LensCorruptions rankings change materially.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Fourier-optics relation between the pupil function and the PSF used to generate kernels from the wavefront."},{"cited_title":"Lens-Designs","cited_arxiv_id":null,"evidence_quote":"The public lens-design database from which the 718 (later 100) nominal lens prescriptions were drawn and curated."}],"review_version":1}