{"id":"3cfe4a8a-ff5f-4e6e-99af-1506cbdfc618","arxiv_id":"2411.09101","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A 42.9M-parameter UNet with a weighted IoU-Dice-cross-entropy loss is compared against a 200M-parameter MaskFormer on iSAID aerial segmentation, reporting mIoU 73.4 and 82.48 respectively.","lead":"This paper compares two ways to segment aerial photos: a small CNN trained from scratch with a custom loss, and a large pre-trained vision transformer fine-tuned for the task. It reports that the large transformer scores higher, but claims the small CNN with the custom loss beats many larger published models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control for the proposed loss: no UNet trained with a standard loss, and the reported 73.4 is a tuned validation-set number compared to prior test-set numbers, so the 'loss boost' and SOTA claims are not supported.","rationale":"The reader's protocol-mismatch concern is valid and material, but the most load-bearing gap is the missing control: even if the evaluation protocol were matched, the paper still would not show that Eq. (5) causes the UNet's performance, because no same-architecture baseline with a standard loss is reported. The validation-set tuning and the unexplained discrepancy between Table I (73.4) and Section IV (peak 81%) reinforce the rejection. I also note that Eq. (4) is printed without the standard negative sign of a cross-entropy term; if implemented literally, minimizing Eq. (5) would push foreground predictions toward 0 and background predictions toward 1, so the implementation code must be checked against the equation. There is no formal verification or reproducible baseline to offset these issues, so the reader's REJECT verdict should stand unchanged.","tokens_in":11091,"tokens_out":8702,"duration_ms":124483,"concrete_test":"Re-train the same UNet architecture and training pipeline described in Section III with a plain cross-entropy or Dice loss (without Eq. 5), using identical augmentation, optimizer, epochs, and foreground-only validation metric, repeated over at least three seeds. If the standard-loss baseline reaches the same validation mIoU as the reported 73.4, or the gap is within run-to-run variance, the central claim that the combined loss drives the boost fails. In addition, re-evaluate both models on the official test set or a held-out split to remove the validation-vs-test comparison issue.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the headline claim that Eq. (5) significantly boosts the CNN, the paper must show that the UNet's 73.4 mIoU is caused by the combined loss. It never trains the same 42.9M UNet with a standard cross-entropy, Dice, or IoU loss under the identical augmentation, optimizer, gradient-clipping, and evaluation protocol, so the gain cannot be assigned to Eq. (5); architecture, data augmentation, or tuning could explain it. The only evidence for the loss is one hand-tuned configuration whose lambda values were selected by trial-and-error on the validation set, and the reported 73.4 is a validation-set mIoU over 15 foreground classes with background excluded. That number is then compared to test-set mIoU of prior works; Section IV's assertion that previous evaluations also ignore background is unsupported. Even internally, Table I reports 73.4 for the UNet while Section IV reports a peak of 81% and stabilization around 78% from Fig. 4. Without a baseline and without protocol reconciliation, neither the causal loss claim nor the 'beats larger models' comparison is established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a heuristic comparison between a custom 42.9M-parameter UNet CNN and a fine-tuned MaskFormer (Swin-L, ~200M parameters) on the iSAID remote-sensing semantic segmentation benchmark. The authors propose a weighted combined loss function (Equation 5) that sums IoU loss, Dice loss, and weighted cross-entropy loss, with weights λ_iou=0.8, λ_dice=1, and λ_ce=10. They report validation-set mIoU values of 73.4 for the UNet and 82.48 for the MaskFormer model, compare these against published test-set results, and claim that the combined loss significantly boosts the CNN's performance. The paper also compares inference time and parameter counts of the two models.","tokens_in":11293,"tokens_out":4981,"duration_ms":53261,"significance":"If the central claims were properly supported, the paper would offer a practically useful result: a relatively small CNN with a tuned loss approaching or exceeding larger ViT-based models on a difficult remote-sensing segmentation benchmark, with efficiency advantages. The manuscript provides a public code repository, describes the data augmentation pipeline in algorithmic detail, and reports parameter counts, FLOPs, and inference times, which are useful for reproducibility. However, the main empirical claims are not supported by the experiments as reported. The absence of a baseline UNet trained with a standard loss means the effect of the proposed loss is not identified, and the comparison to prior work is based on validation-set numbers versus test-set numbers without protocol reconciliation. The internal inconsistency between Table I and Figure 4 further undermines confidence in the reported metrics. These issues are load-bearing for the paper's headline conclusions, so the significance of the work is not established in its current form.","major_comments":[{"comment":"The central claim that the proposed combined weighted loss significantly boosts the UNet's performance is not supported because no control experiment is reported. The paper never trains the same 42.9M UNet architecture with a standard cross-entropy, Dice, or IoU loss under the identical augmentation, optimizer, gradient-clipping, and evaluation protocol. Without such a baseline, the reported 73.4 mIoU cannot be attributed to Equation (5); architecture choices, augmentation, gradient accumulation, mixed precision, or hyperparameter tuning could equally explain the result.","section":"Section IV and Section III-D (Equation 5)"},{"comment":"The comparison against prior state-of-the-art models is not valid as stated. The paper's mIoU values are computed on the validation set with the background class excluded (Section III-A: 'We only consider the foreground category while calculating our validation metrics scores'; Section IV: 'Our UNet CNN model's metrics on the validation set surpassed...'), while the cited results (e.g., AerialFormer, RingMo) are test-set results from their respective papers. Section IV asserts that 'the background class is not taken into consideration in any, ours or the previous, of the evaluation' without providing evidence for this claim. Unless the exact evaluation protocol (dataset split, background handling, class averaging, and whether previous works used the same validation set) is reconciled, the headline comparisons—UNet 73.4 versus AF-B 69.3, and MaskFormer 82.48 versus the cited models—are not established.","section":"Section IV, Table I, Section III-A"},{"comment":"There is an internal inconsistency in the reported UNet accuracy. Table I reports an mIoU of 73.4 for 'Ours (CNN)', while Section IV states that the UNet 'reached the peak of 81% and stabilized at around 78% during training over all classes' and refers to Figure 4. The paper must clarify whether Table I reports the final-epoch validation mIoU over the 15 foreground classes only, and why Figure 4 shows different peak and final values. As written, the two numbers cannot both describe the same metric, and this discrepancy undermines the reliability of the reported results.","section":"Section IV versus Table I"},{"comment":"The loss-function hyperparameters (λ_iou=0.8, λ_dice=1, λ_ce=10) and the class weight β=0.15 are selected by trial and error on the same validation set used for reporting the headline results, and no ablation or sensitivity analysis is provided. Since Equation (5) is the paper's main contribution, the lack of any ablation leaves both the contribution and the risk of validation-set overfitting unquantified. A minimal analysis varying each weight while holding the others fixed is needed to demonstrate that the loss components matter and that the specific values are not the sole driver of the reported mIoU.","section":"Section III-D and Section III-F"}],"minor_comments":[{"comment":"Several citations are unresolved placeholders, including '[?]' after 'Papers with Code' and after 'SegFormer' and 'MaskFormer' in the related-work discussion; these need to be filled in before submission.","section":"Section II"},{"comment":"The gradient clipping update rule is typeset incorrectly; it should read: if ||ĝ|| > threshold, set ĝ ← (threshold / ||ĝ||) · ĝ.","section":"Algorithm 2"},{"comment":"The description of β is ambiguous: the text says 'β is the weight hyperparameter (0.15 for unlabelled class and 1 for the rest)' but then says 'We choose the minimal value of 0.15 for β.' Please clarify whether β is a scalar or a per-class weight vector, and specify the per-pixel formula when β=1 for foreground classes; as written, β=1 in Equation (4) makes the second term vanish for foreground pixels, which should be stated explicitly.","section":"Section III-D, Equation (4)"},{"comment":"The sentence 'One of the main pitfalls of such datasets is the background class [11]' cites a qualitative health research paper (Hinds et al.) that is not relevant to background-class handling in segmentation; an appropriate citation on class-imbalance or background dominance in remote-sensing datasets should be provided.","section":"Introduction"},{"comment":"The phrase 'This correlation fgvrther validates the experimental design' contains a typo ('fgvrther' should be 'further'), and the causal wording 'validates the experimental design of the combined loss' overstates what visual inspection of qualitative segmentation maps can establish.","section":"Section IV"},{"comment":"The MaskFormer row lists 200M parameters, while the text says 'resulting to approximately 200M parameters'; please report the exact parameter count and state clearly how many parameters are trainable versus frozen, since the fine-tuning protocol is part of the comparison.","section":"Table I and Section III-C"}],"recommendation":"reject","confidential_remarks":"The manuscript addresses a relevant empirical question, but the two main claims—the effectiveness of the proposed loss and the favorable comparison against prior models—are not supported by the evidence as presented. The missing control condition and the validation-versus-test-set comparison are not local issues; they require new experiments and a re-evaluation of the results. The internal inconsistency between Table I and Figure 4 also needs resolution. I would encourage the authors to resubmit after adding a proper standard-loss baseline, performing a loss-weight ablation, and reporting results on the official test set with a clearly described evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on Dahal et al. The paper does something real: it trains a 42.9M UNet and a 200M MaskFormer on iSAID, reports class-wise IoU, inference times, and ships code. The empirical comparison, taken as a measurement, is useful for anyone working on remote sensing segmentation. The loss function is a weighted sum of soft-IoU, Dice, and weighted cross-entropy — known pieces, no new math there. The claimed novelty is the combination and the claimed boost to the CNN.\n\nThe soft spots are exactly where the reader puts them, and they are load-bearing. First, there is no UNet trained with a standard loss under the same protocol. Without that baseline, the 73.4 mIoU cannot be attributed to Eq. (5). The weights were hand-tuned on the validation set, and the same set is used for the headline comparison. Second, Table I compares validation mIoU on 15 foreground classes to test-set numbers from prior papers. Section IV asserts that prior evaluations also exclude background, but no citation or protocol evidence is given. That makes the \"beats AF-B and RingMo\" claim unverifiable. Third, there is an internal inconsistency: Table I reports 73.4 for the UNet, but Section IV describes a peak of 81% stabilizing around 78%. That might be training curves or a different metric, but the paper doesn't say, and it undermines confidence.\n\nI also note the paper's own statement: \"Our UNet CNN model's metrics on the validation set surpassed the performance of similarly comparable references on the test set.\" That is an honest admission, but it also concedes the central comparison is apples-to-oranges. The abstract's claim that the novel loss \"significantly boosts\" the CNN is not supported by the experiments as designed.\n\nCredit where due: the paper is clearly written (aside from a few typos), the literature review is active, and the authors are transparent about some limitations, like the augmentation cost and the background issue. They don't hide that the weights were trial-and-error. That honesty is worth something.\n\nBottom line: the measurements could be salvageable, but the paper as written does not establish its main claim. I would not cite it. I'd still send it to peer review, because the flaws are identifiable and fixable with extra experiments — a proper baseline and a test-set evaluation — but it needs major revision, and if the authors can't provide the baseline, the \"boost\" claim should be dropped.\n\nFor the reading group, it's a decent case study in experimental design pitfalls.\n\nRecommendation: engage with it as a reviewer, but be prepared to require substantial new experiments.","headline":"Useful empirical measurements undermined by a missing baseline and protocol mismatch; the 'loss boost' claim does not hold as written.","tokens_in":11922,"tokens_out":3163,"would_cite":false,"duration_ms":33718,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 42.9M-parameter UNet with a three-part weighted loss reaches 73.4 mIoU on iSAID, surpassing larger transformer models.","keywords":["Vision Transformer","semantic segmentation","remote sensing","iSAID","UNet","combined weighted loss","mean Intersection over Union","transfer learning"],"falsifier":"Evaluate the trained UNet on the official iSAID test set, or re-evaluate the cited baselines on the same validation split with the same background-excluded protocol; if the matched mIoU drops below the 69.3 reported for the 113.8M-parameter baseline, the central claim fails.","tokens_in":10820,"feed_emoji":"🛰️","tokens_out":13043,"duration_ms":104321,"temperature":0.7,"pith_summary":"This paper reports an empirical comparison between a from-scratch UNet CNN and a transfer-learned MaskFormer vision transformer on the iSAID remote-sensing segmentation benchmark. It argues that the key driver of performance is not the encoder architecture alone but the loss function: a weighted sum of an IoU loss, a Dice loss, and a weighted cross-entropy term. With that combined loss, a 42.9M-parameter UNet reaches 73.4 mIoU on the validation set, a score the paper says surpasses published results from larger models such as the 113.8M-parameter AerialFormer-B (69.3 mIoU) and a 100M-parameter plain ViT (67.2 mIoU). The 200M-parameter MaskFormer baseline still scores higher in raw accuracy (82.48 mIoU), but at roughly five times the parameters and slower inference. A sympathetic reader would take the paper's central claim to be that a small CNN with a carefully weighted loss can approach or beat much larger attention-based models on this task, which matters for deployment on platforms with limited compute.","feed_headline":"Tuned loss lifts a 42.9M CNN past 113.8M ViTs","feed_subtitle":"A three-part weighted loss gives a small UNet 73.4 mIoU on iSAID aerial segmentation, beating far larger models.","key_machinery":"The load-bearing object is the combined weighted loss function in Eq. (5): $$L = \\lambda_{\\mathrm{iou}} L_{\\mathrm{iou}} + \\lambda_{\\mathrm{dice}} L_{\\mathrm{dice}} + \\lambda_{\\mathrm{ce}} L_{\\mathrm{ce}}$$ where $L_{\\mathrm{iou}}$ replaces the non-differentiable intersection and union with multiplication and addition, $L_{\\mathrm{dice}}$ uses the generalized Dice overlap, and $L_{\\mathrm{ce}}$ is a weighted cross-entropy with background weight $\\beta = 0.15$ and foreground weight 1. The weights $\\lambda_{\\mathrm{iou}} = 0.8$, $\\lambda_{\\mathrm{dice}} = 1$, and $\\lambda_{\\mathrm{ce}} = 10$ were chosen by trial and error. This loss is what the paper credits with letting the small UNet generalize on background pixels and small objects, while the UNet itself is a standard four-skip-connection encoder-decoder with 42.9M parameters. The MaskFormer baseline uses the same Dice-plus-focal loss from its reference implementation.","core_discovery":"On the paper's own terms, the central discovery is that the proposed combined weighted loss function—$L = \\lambda_{\\mathrm{iou}} L_{\\mathrm{iou}} + \\lambda_{\\mathrm{dice}} L_{\\mathrm{dice}} + \\lambda_{\\mathrm{ce}} L_{\\mathrm{ce}}$ with weights 0.8, 1, and 10—lets a generic 42.9M-parameter UNet, trained from scratch with mixed precision and gradient accumulation, reach 73.4 mIoU on the iSAID validation set while excluding the background class from evaluation. The same table reports per-class IoU scores that beat the cited AerialFormer-B, RingMo, and plain-ViT baselines on most categories, and the paper notes that the MaskFormer ViT (about 200M parameters) reaches 82.48 mIoU. The authors interpret this as evidence that the combined loss improves generalization and background handling, that the UNet is more inference-efficient (0.19s versus 0.29s on six images), and that the parameters needed for effective remote-sensing segmentation do not need to exceed 50M. They also state explicitly that background pixels are not counted in any of the evaluations, theirs or the prior works they compare against.","pith_inferences":["The headline comparison is only as strong as its evaluation protocol: the 73.4 mIoU is a validation-set number with the background class excluded, whereas the cited works' numbers may come from the official test server with different background handling; a same-protocol rerun could shift the ranking.","The very low background weight ($\\beta = 0.15$) in the cross-entropy term suggests the loss is engineered to ignore the unlabeled majority class; on fully labeled datasets with no background class, the weight balance would likely need to be re-derived.","A natural testable extension is to apply the same combined loss to other aerial or satellite segmentation benchmarks with different class imbalance, to see whether the gain is specific to iSAID or generalizes.","Because the three loss weights were selected by trial and error, the paper leaves open whether a small grid search or a learned weighting would yield further gains on the same CNN."],"forward_implications":["If the 73.4 mIoU holds under matched evaluation, a 42.9M-parameter CNN can surpass published iSAID results from models two to five times larger, showing that loss design can compensate for a much smaller architecture.","The MaskFormer ViT's higher raw score (82.48 mIoU) at 200M parameters and slower inference shapes the trade-off as roughly five times the parameters for about 9 mIoU points.","The paper's inference measurements (0.19s versus 0.29s on six images) place the CNN ahead for time-critical applications such as UAV and ground-vehicle segmentation.","The paper's conclusion that a workable remote-sensing segmentation model does not need more than 50M parameters follows directly if the 73.4 mIoU result is reproducible under matched conditions."],"supporting_citations":[{"why":"Supplies the iSAID dataset, its 15 foreground classes plus background, and the train/validation/test split that defines the task.","marker":"[32]"},{"why":"Defines the MaskFormer ViT baseline used for transfer learning, including its Dice-plus-focal loss formulation.","marker":"[2]"},{"why":"Provides the AerialFormer-T/S/B state-of-the-art results, including the 113.8M-parameter AF-B at 69.3 mIoU that the UNet is claimed to beat.","marker":"[9]"},{"why":"Provides the 100M-parameter plain ViT baseline at 67.20 mIoU that the paper compares against.","marker":"[24]"},{"why":"Provides the RingMo remote-sensing foundation-model baseline at 67.2 mIoU in the comparison table.","marker":"[22]"},{"why":"Supplies the SAMRS UperNet-based CNN and ViT results (62.54 and 66.26 mIoU) the paper claims to improve upon.","marker":"[25]"},{"why":"Gives the IoU loss whose differentiable intersection/union replacement is the $L_{\\mathrm{iou}}$ component of the combined loss.","marker":"[35]"},{"why":"Gives the generalized Dice overlap that forms the $L_{\\mathrm{dice}}$ component of the combined loss.","marker":"[21]"},{"why":"Provides the cross-entropy loss formulation with class weighting that forms the $L_{\\mathrm{ce}}$ component.","marker":"[16]"}],"fun_headline_variants":["Weighted loss lifts 42.9M UNet past ViTs on iSAID","Three-part loss gives small CNN 73.4 mIoU, beating ViT baselines","Loss engineering: tiny UNet outperforms ViTs in aerial segmentation","73.4 mIoU from 42.9M CNN: tuned loss trumps ViT size","Small CNN beats ViT baselines with fused loss on iSAID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the paper's validation-set mIoU, computed over 15 foreground classes only, is directly comparable to the test-set mIoU of the cited models, even though the cited works may use a different evaluation protocol or include the background class.","fun_headline_variants_meta":{"raw":{"variants":["Weighted loss lifts 42.9M UNet past ViTs on iSAID","Three-part loss gives small CNN 73.4 mIoU, beating ViT baselines","Loss engineering: tiny UNet outperforms ViTs in aerial segmentation","73.4 mIoU from 42.9M CNN: tuned loss trumps ViT size","Small CNN beats ViT baselines with fused loss on iSAID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000423,"raw_usage":{"total_tokens":2247,"prompt_tokens":1094,"completion_tokens":1153,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":1042}},"tokens_in":710,"tokens_out":1153,"duration_ms":11851,"temperature":1.0,"reasoning_tokens":1042,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:01:37.590506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the trained UNet on the official iSAID test set, or re-evaluate the cited baselines on the same validation split with the same background-excluded protocol; if the matched mIoU drops below the 69.3 reported for the 113.8M-parameter baseline, the central claim fails.","supporting_citations":[{"cited_title":"iSAID: A Large-scale Dataset for Instance Seg- mentation in Aerial Images,","cited_arxiv_id":null,"evidence_quote":"Supplies the iSAID dataset, its 15 foreground classes plus background, and the train/validation/test split that defines the task."},{"cited_title":"Per-Pixel Classification is Not All You Need for Semantic Segmentation,","cited_arxiv_id":null,"evidence_quote":"Defines the MaskFormer ViT baseline used for transfer learning, including its Dice-plus-focal loss formulation."},{"cited_title":"AerialFormer: Multi-Resolution Transformer for Aerial Image Segmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the AerialFormer-T/S/B state-of-the-art results, including the 113.8M-parameter AF-B at 69.3 mIoU that the UNet is claimed to beat."},{"cited_title":"Advancing plain vision transformer toward remote sensing foundation model,","cited_arxiv_id":null,"evidence_quote":"Provides the 100M-parameter plain ViT baseline at 67.20 mIoU that the paper compares against."},{"cited_title":"RingMo: A Remote Sensing Foundation Model With Masked Image Modeling,","cited_arxiv_id":null,"evidence_quote":"Provides the RingMo remote-sensing foundation-model baseline at 67.2 mIoU in the comparison table."},{"cited_title":"Iou loss for 2d/3d object detection,","cited_arxiv_id":null,"evidence_quote":"Gives the IoU loss whose differentiable intersection/union replacement is the $L_{\\mathrm{iou}}$ component of the combined loss."},{"cited_title":"Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,","cited_arxiv_id":null,"evidence_quote":"Gives the generalized Dice overlap that forms the $L_{\\mathrm{dice}}$ component of the combined loss."},{"cited_title":"Cross-entropy loss functions: Theoretical analysis and applications,","cited_arxiv_id":null,"evidence_quote":"Provides the cross-entropy loss formulation with class weighting that forms the $L_{\\mathrm{ce}}$ component."}],"review_version":1}