{"id":"0dd0ae01-0114-44f7-8868-810a904ff365","arxiv_id":"2412.12565","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A PBVS 2024 solution report combining DINOv2, Lee filtering, SAR-to-EO translation, and undersampling achieves 9th place (21.45% accuracy) without ablations or code.","lead":"This paper describes a competition entry for the PBVS 2024 SAR classification challenge: a DINOv2-based pipeline with Lee filtering, SAR-to-EO translation, and Tomek/NearMiss undersampling that scored 21.45% accuracy and 0.56 AUC, placing 9th. It is a short report with no ablations or code to support the claimed benefits of its components.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation or baseline is reported, so the paper's central claim that Lee filtering, SAR-to-EO translation, Tomek Links, NearMiss-3, and ensembling improve long-tail SAR classification is unsupported; the 9th-place score cannot be attributed to the proposed pipeline.","rationale":"Good-faith reading: this is a concise competition report. The only externally verifiable claim is the competition outcome, and that is credible. The problem is that the paper's scientific contribution is not the leaderboard position but the claim that a specific set of techniques is effective for the stated challenges. That claim requires attribution: each design choice must be shown to contribute. The reader's weakest_assumption identifies exactly this. I agree. I also note a concrete internal inconsistency: the Introduction explicitly promises clustering as a component, but the methodology never uses it, which reinforces that the narrative is not tightly coupled to the experiments. The concrete test would settle attribution with one controlled comparison. Because the central claim is unsupported, the REJECT verdict stands; no adjustment is needed.","tokens_in":4647,"tokens_out":3939,"duration_ms":32388,"concrete_test":"On a held-out validation split of the PBVS 2024 dataset, rerun the pipeline with the three-channel input replaced by original SAR only, and with Tomek Links and NearMiss-3 disabled (e.g., uniform random undersampling), keeping DINOv2 feature extraction and the KNN classifier identical. If this baseline achieves a total score within about 0.02 of the reported 0.30, the proposed components are not shown to help; if it clearly underperforms, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central methodological claim—that the proposed preprocessing and sampling pipeline improves classification under extreme long-tail and SAR/EO domain shift—rests entirely on the final competition score (21.45% accuracy, 0.30 total score, 9th place). No experiment isolates any component. Section 2.1 introduces a three-channel input (original SAR, Lee-filtered SAR, Pix2PixHD-translated EO) but provides no comparison against single-channel or two-channel inputs; the claim that concatenation 'mitigates' shape distortion is asserted, not tested. Section 2.2 applies Tomek Links and NearMiss-3 to DINOv2 features before training KNN classifiers, yet no baseline with uniform sampling, no resampling, or a single classifier is reported. The reported score could therefore be produced entirely by a pretrained DINOv2 feature extractor plus KNN, with the added components being neutral or even harmful. Additionally, the Introduction promises 'clustering' ('Clustering helps identify representative samples'), but no clustering algorithm appears anywhere in Sections 2 or 3; the method as described is internally inconsistent. Without ablations or a baseline, the causal claims in the Conclusion ('these balanced subsets provided an optimal foundation', 'the ensemble further boosts robustness') are unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the authors' solution for the PBVS 2024 Multi-modal Aerial View Imagery Challenge Classification (MAVIC-C) SAR classification task. The proposed pipeline consists of (i) preprocessing SAR inputs with a Lee filter and Pix2PixHD-based SAR-to-EO translation, then concatenating original SAR, denoised SAR, and synthetic translated EO into a three-channel input; (ii) self-supervised representation learning with DINOv2; (iii) long-tail handling by Tomek Links and NearMiss-3 undersampling of DINOv2 features to create seven balanced subsets; and (iv) ensembling K-nearest-neighbor classifiers trained on each subset. The authors report official test-set results of 21.45% accuracy, 0.56 AUC, and a total score of 0.30, placing 9th in the competition, and attribute this outcome to their proposed components.","tokens_in":4993,"tokens_out":8087,"duration_ms":70576,"significance":"If the claimed benefits were supported by evidence, the combination of SAR denoising, SAR-to-EO translation, and instance-based undersampling with self-supervised DINOv2 features would be of practical interest for extreme long-tail SAR classification. The use of an external competition test set is a genuine strength: the reported score avoids training/evaluation circularity. However, the manuscript contains no baseline, no ablation, and no sensitivity analysis, so the contribution of any individual component is unverified. The reported result is also modest—9th place, with total score 0.30 compared to 0.49 for the winning team—so the practical significance of the pipeline is not demonstrated as submitted.","major_comments":[{"comment":"The central empirical claim of the paper—that Lee filtering, SAR-to-EO translation, three-channel concatenation, Tomek Links, NearMiss-3, and ensembling improve long-tail SAR classification—is unsupported. The only quantitative result is the official competition score reported in Section 3 and Table 1; no comparison is made against a plain DINOv2-backed KNN baseline, single-channel or two-channel inputs, no-resampling training, or a single KNN classifier. Without such ablations, the final score could be produced by the pretrained DINOv2 feature extractor and KNN alone, and the causal statements in Section 4 ('these balanced subsets provided an optimal foundation', 'the ensemble further boosts robustness') are unverifiable. Please add ablations on a validation split, or on the frozen challenge test set if additional submissions are permitted, isolating each proposed component.","section":"Sections 2 and 3"},{"comment":"The Introduction promises a clustering component ('we combine clustering and self-supervised learning', 'Clustering helps identify representative samples'), but no clustering algorithm appears in Sections 2.2 or 3. Tomek Links and NearMiss-3 are undersampling and instance-selection techniques, not clustering, and KNN is a classifier, not a clustering procedure. This internal inconsistency must be resolved: either add a clustering step to the pipeline or revise the Introduction to describe the actual undersampling procedure.","section":"Section 1 vs. Section 2.2"},{"comment":"The claim that concatenating original SAR, Lee-filtered SAR, and SAR-to-EO translated images into a three-channel input 'mitigates' the shape distortion introduced by denoising and translation is asserted without experiment. Moreover, although the challenge provides real EO images for training, the paper never describes using them in classification; only synthetic EO from SAR-to-EO translation appears in the input. The role of each modality and the benefit of multi-channel concatenation should be clarified and empirically tested.","section":"Section 2.1"}],"minor_comments":[{"comment":"The caption should identify which panel is which (original SAR, Lee-filtered SAR, translated EO, reference EO); the current caption '(d)(c)(b)(a)' is confusing.","section":"Figure 3"},{"comment":"The paper says DINOv2 was 'trained' on the entire dataset but also that it was 'initialized with pre-trained weights from DINOv2'; specify whether this is continued self-supervised pretraining or fine-tuning, and report training epochs, loss, and compute.","section":"Section 3"},{"comment":"The construction of the 'N balanced subsets' is not described; explain how Tomek Links and NearMiss-3 are applied to high-dimensional DINOv2 features and how the seven subsets are generated (e.g., bootstrap or disjoint partitions).","section":"Section 2.2"},{"comment":"The total score formula used to rank teams is not defined; without it, the relation between accuracy, AUC, and the final ranking is unclear.","section":"Table 1"},{"comment":"The text alternates between 'DINOv2' and 'Dinov2'; please use consistent capitalization.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper is a competition report rather than a self-contained research contribution. My recommendation of reject is driven by the absence of any baseline or ablation; to make this publishable the authors would need to add a substantial experimental section comparing each component, which goes beyond the current manuscript's scope. If such experiments are available, a resubmission would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a genuinely reported 9th-place competition score, but the paper never shows that the proposed pipeline produced it. No ablation, no baseline, just the final number and a table.\n\nWhat is actually new: not much individually. The pipeline combines DINOv2 features, Lee filtering, Pix2PixHD translation, Tomek Links, NearMiss-3, and KNN ensembling, plus the three-channel concatenation trick. All components are known; the specific combination and the score on the PBVS 2024 test set are new. The paper is honest about the score, and the description of the pipeline is clear enough to reproduce if you had the code (which is not provided).\n\nSoft spots: the central methodological claim—that these components help under extreme long-tail and domain shift—is untested. Section 2.1 asserts that concatenating original SAR, denoised SAR, and synthetic EO mitigates shape distortion; no comparison against one- or two-channel inputs. Section 2.2 asserts that Tomek Links and NearMiss-3 balance the classes and improve rare-category performance; no comparison against no resampling or a single classifier. The score alone could come almost entirely from a pretrained DINOv2 with KNN, with the rest neutral or even harmful. The conclusion's causal language ('these balanced subsets provided an optimal foundation', 'the ensemble further boosts robustness') is not supported by the experiments. Also, the introduction promises clustering as part of the approach, but no clustering appears in the methodology; that's an internal inconsistency.\n\nOne more minor issue: free parameters such as K=3, 7 subsets, resize to 56x56 are given without any sensitivity analysis. That's minor for a competition report, but it adds to the sense that the paper is a high-level summary rather than a rigorous study.\n\nWho is this for? People interested in what the PBVS participants did, or in the specific score. It has value as a workshop-level write-up. As a research paper, it is too thin: the load-bearing claims about the method are not backed by experiments. My recommendation: desk reject for a serious venue; if the workshop wants a record of participation, it could appear as an extended abstract, but it should not go to peer review as a full paper. A referee would have to ask for ablations that would essentially be a new paper.","headline":"Honest competition write-up with a real external score, but the paper never shows the pipeline caused that score.","tokens_in":5463,"tokens_out":2712,"would_cite":false,"duration_ms":22425,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In an extreme long-tail SAR classification challenge, a two-stage self-supervised pipeline with undersampled KNN ensembles reached 21.45% accuracy and 9th place.","keywords":["SAR classification","long-tail learning","self-supervised learning","DINOv2","SAR-to-EO translation","Lee filter","undersampling","KNN ensemble"],"falsifier":"Run the same DINOv2-KNN ensemble on single-channel original SAR images, then add the Lee-filtered channel and then the translated channel, and compare total scores on the PBVS 2024 validation or test set; if the score does not drop when the extra channels are removed, the claim that the multimodal preprocessing helps is falsified. In parallel, compute per-class accuracy for the smallest classes with and without Tomek Links and NearMiss-3; absence of improvement there would falsify the balancing claim.","tokens_in":4483,"feed_emoji":"🛰️","tokens_out":10078,"duration_ms":77667,"temperature":0.7,"pith_summary":"The paper confronts a vehicle-classification task in which the largest class has 364,291 training samples and the smallest has 353, an imbalance of more than 1000 to 1 that defeats standard long-tail remedies. It argues that the workable recipe is a two-stage design: first train a self-supervised vision transformer (DINOv2) on the full unlabeled set so that rare-class structure is not sacrificed, then balance the feature space with Tomek Links and NearMiss-3 and combine the predictions of several K-nearest-neighbor classifiers, each trained on a balanced subset. To bridge the radar-optical domain gap, each input is a three-channel image made of the original SAR view, a Lee-filtered (despeckled) SAR view, and a SAR-to-EO translated view. On the held-out test set this system posted 21.45% accuracy, 0.56 AUC, and a total score of 0.30, ranking 9th in the challenge. The paper's contribution is the specific combination of these ingredients for an extreme long-tail setting, rather than a claim that any individual stage is sufficient.","feed_headline":"Self-supervised pipeline scores 21.45% on extreme long-tail SAR","feed_subtitle":"A two-stage DINOv2 plus undersampling design reached 9th in the PBVS 2024 SAR classification challenge.","key_machinery":"The central mechanism is a two-stage pipeline built around a frozen self-supervised vision transformer (DINOv2). Stage one learns representations on the entire dataset without labels so that head-class bias does not enter the feature extractor; stage two uses those features to construct balanced training subsets via Tomek Links and NearMiss-3 and trains an ensemble of distance-based KNN classifiers (K=3) on them. The input representation is equally load-bearing: a three-channel image obtained by concatenating the original SAR, a Lee-filtered (despeckled) SAR, and a synthetic EO image produced by Pix2PixHD, so the network can exploit complementary views without explicit multimodal fusion. Each piece is designed to counter one of the two stated failure modes, the extreme class imbalance and the SAR/EO domain disparity.","core_discovery":"The central claim is that a severe long-tail SAR classification problem can be handled without ever training a supervised classifier on raw imbalanced data, and that the radar-optical domain gap can be folded into the input representation rather than into a custom fusion network. DINOv2 is first trained self-supervised on all 455,600 samples; then, in the frozen feature space, Tomek Links removes majority samples that sit close to minority samples and NearMiss-3 selects the most ambiguous remaining samples, producing several balanced subsets. A KNN classifier with K=3 is trained on each subset and the outputs are ensembled. In parallel, every SAR image is despeckled with a Lee filter and translated into an EO-style image with Pix2PixHD, and the original, filtered, and translated views are concatenated into a three-channel input used at both training and inference. The authors report that this pipeline achieved 21.45% accuracy, 0.56 AUC, and a total score of 0.30, placing 9th in the PBVS 2024 MAVIC-C.","pith_inferences":["Inference: The paper does not report per-class accuracy, but its own logic predicts that the undersampling and ensembling steps earn their keep on the smallest classes; a reader could test this by computing rare-class accuracy with and without Tomek Links and NearMiss-3.","Inference: Since no ablation is given, the natural follow-up experiment is to run the same DINOv2-KNN ensemble on the original SAR channel alone and then add the Lee-filtered and translated channels one at a time; if the total score does not move, the multimodal preprocessing is not the active ingredient.","Inference: The dependence on Pix2PixHD for SAR-to-EO translation adds a generation step at inference time, which could be a practical bottleneck; replacing it with lighter synthesis or direct feature-space alignment is an obvious extension of the approach.","Inference: The same two-stage recipe could be transported to other paired-sensor tasks (for instance optical-to-infrared) where one view is noisy and class counts differ by orders of magnitude, though the paper itself does not claim this."],"forward_implications":["If the pipeline works as described, the self-supervised feature extractor can be trained on the full unlabeled dataset before any balancing, so rare-class information is not discarded during pretraining.","The three-channel input, composed of original, Lee-filtered, and SAR-to-EO translated views, is used at both training and inference, giving the model access to both modalities without a separate fusion network.","The combination of Tomek Links, NearMiss-3, and ensembled KNN classifiers is the paper's proposed way to build a balanced classifier in an extreme 1000-to-1 long-tail setting.","The method's final numbers on the PBVS 2024 test set are 21.45% accuracy, 0.56 AUC, and a total score of 0.30, ranking 9th."],"supporting_citations":[{"why":"Defines the PBVS 2024 MAVIC-C challenge, its dataset, and the scoring metric on which the reported result is measured.","marker":"[6]"},{"why":"Supplies DINOv2, the self-supervised vision transformer whose features carry the entire two-stage learning and balancing design.","marker":"[7]"},{"why":"Supplies Pix2PixHD, the conditional GAN used to translate SAR images into synthetic EO images for the third input channel.","marker":"[12]"},{"why":"Supplies the refined Lee filter used to despeckle SAR images before concatenation.","marker":"[13]"},{"why":"Provides the Vision Transformer architecture on which DINOv2 is based, grounding the choice of the feature extractor.","marker":"[2]"}],"fun_headline_variants":["Self-supervised SAR wins 9th with 21.45% on extreme long-tail","No supervised training: SAR classification via DINOv2 and smart sampling","DINOv2 plus undersampling cracks extreme long-tail SAR challenge","Self-supervised pipeline hits 21.45% in harsh SAR tail distribution","Long-tail SAR tamed: self-supervised learning and sampling to 9th"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes each pipeline component helps, because it reports no experiment that removes any single component to check whether the gains actually come from the combination.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised SAR wins 9th with 21.45% on extreme long-tail","No supervised training: SAR classification via DINOv2 and smart sampling","DINOv2 plus undersampling cracks extreme long-tail SAR challenge","Self-supervised pipeline hits 21.45% in harsh SAR tail distribution","Long-tail SAR tamed: self-supervised learning and sampling to 9th"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3085,"prompt_tokens":1031,"completion_tokens":2054,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":1952}},"tokens_in":647,"tokens_out":2054,"duration_ms":13395,"temperature":1.0,"reasoning_tokens":1952,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:56:41.228829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same DINOv2-KNN ensemble on single-channel original SAR images, then add the Lee-filtered channel and then the translated channel, and compare total scores on the PBVS 2024 validation or test set; if the score does not drop when the extra channels are removed, the claim that the multimodal preprocessing helps is falsified. In parallel, compute per-class accuracy for the smallest classes with and without Tomek Links and NearMiss-3; absence of improvement there would falsify the balancing claim.","supporting_citations":[{"cited_title":"Multi-modal aerial view image challenge: Sar classification","cited_arxiv_id":null,"evidence_quote":"Defines the PBVS 2024 MAVIC-C challenge, its dataset, and the scoring metric on which the reported result is measured."},{"cited_title":"High-resolution image syn- thesis and semantic manipulation with conditional gans","cited_arxiv_id":null,"evidence_quote":"Supplies Pix2PixHD, the conditional GAN used to translate SAR images into synthetic EO images for the third input channel."},{"cited_title":"Sar image despeckling using refined lee filter","cited_arxiv_id":null,"evidence_quote":"Supplies the refined Lee filter used to despeckle SAR images before concatenation."}],"review_version":1}