{"id":"bba40111-7788-45b2-9aee-27885e8621a4","arxiv_id":"2505.10737","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning BirdDetector with stronger augmentations and SAHI inference improved average F1 for Salvin's albatross detection on Bounty Islands from 0.66 zero-shot to 0.75.","lead":"This paper tests whether adapting a general-purpose bird-spotting program with extra image augmentation improves automated counts of Salvin's albatrosses from drone photos of New Zealand's Bounty Islands. It reports that the adapted program scores higher on a detection-accuracy metric, rising from 0.66 to 0.75, which could make monitoring a vulnerable seabird faster and cheaper.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IoU threshold of 0.1 makes F1 gains hard to interpret for actual detection quality.","rationale":"The reader identified the pseudo-boxes plus permissive IoU as the weakest assumption, and I agree that this is the most load-bearing vulnerability in the empirical claim. The paper's entire quantitative case is a single table of F1 values; the metric combines detection and localization, but the localization component is nearly disabled by the IoU threshold of 0.1 and fixed-size pseudo-boxes. Section 2 admits annotations were occasionally offset from bird centers, and Section 3.2 explicitly says the threshold 'was set very low to account for possible inaccurate boxes generated from the annotated points.' That is an honest admission but also a direct warning that the metric is tolerant of poor localization. However, I do not think this alone requires rejection: the paper is transparent about its threshold choices, the experiment is internally consistent, and the direction of the effect (stronger augmentation improves F1) could plausibly survive a stricter threshold because augmentation is known to help generalization. To settle it, the authors should re-run the same detections under IoU 0.3 and 0.5; since all thresholds are post-hoc, this requires no retraining. I also note the weaker but real issues: no error bars from single runs, no code/data release, and evaluation only on annotated tiles, which makes the counting claim conditional. But the IoU sensitivity is the single most decisive check, so the verdict stays CONDITIONAL rather than moving to UNCHANGED or REJECT.","tokens_in":8388,"tokens_out":1644,"duration_ms":13719,"concrete_test":"Recompute the validation/test F1 for the four configurations in Table 1 at IoU thresholds of 0.1, 0.3, and 0.5 using the same detections and pseudo-boxes, and report per-island and average F1 at each threshold. If the qualitative ordering (fine-tuned with augmentations beats zero-shot SAHI) and the magnitude of the gain (0.75 vs 0.66) survive at IoU=0.5, the central claim is robust; if the gap collapses at IoU=0.5, the paper's headline improvement is an artifact of the permissive matching criterion.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim is that fine-tuning plus stronger augmentations improves detection accuracy, supported by an average F1 gain from 0.6576 (zero-shot SAHI) to 0.7504. But F1 is computed against pseudo-bounding boxes made by centering a fixed 50x50-pixel square on each manual point annotation, and a detection is a true positive if IoU with that pseudo-box is at least 0.1. Because the pseudo-box area is fixed, a true-positive match can be extremely loose: for a 50x50 box of area 2500, a predicted box of area 2500 can be offset by well over half a bird body and still exceed IoU 0.1, and much smaller or larger predicted boxes also qualify easily. The paper itself acknowledges in Section 3.2 that the low IoU threshold was set to account for inaccurate boxes generated from annotated points. The consequence is that the reported F1 values primarily measure whether a detection lands in the broad neighborhood of any manual point, not whether the predicted box localizes the bird. This is especially load-bearing because the headline use case is population counting, where the metric that matters is detection (count) accuracy; if F1 is inflated by permissive matching, the claimed gains from stronger augmentation could shrink or disappear under a stricter threshold such as 0.5, the standard in object detection. A sensitivity check at three IoU thresholds would either confirm the qualitative ordering or reveal that the improvements are largely an artifact of loose matching.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript evaluates the general-purpose BirdDetector model for detecting Salvin's albatrosses in drone orthomosaics of eight Bounty Islands islets. Point annotations from a prior survey are converted to 50x50-pixel pseudo-bounding boxes, yielding 571 supertiles containing 66,635 annotated birds. Four configurations are compared under leave-one-island-out cross-validation: zero-shot without SAHI, zero-shot with SAHI, fine-tuning with the original augmentation schedule, and fine-tuning with stronger augmentations. Detection quality is measured by F1 using an IoU threshold of 0.1 and a confidence threshold of 0.1. The headline result is that the strongest configuration improves average F1 from 0.6576 (zero-shot with SAHI) to 0.7504, and the paper concludes that target-domain fine-tuning and stronger augmentation lead to marked improvements in detection accuracy.","tokens_in":8752,"tokens_out":6093,"duration_ms":61089,"significance":"If the claimed effect is robust, the practical contribution is real: the paper demonstrates a workflow for adapting a general bird detector to a dense, remote seabird colony with modest annotation effort, and it provides per-island generalization results that ecologists can use for monitoring. The evaluation is grounded by the use of an externally pretrained model (BirdDetector), held-out-island evaluation, fixed thresholds that were not tuned on a validation set, and transparent reporting of per-island F1 scores rather than a single pooled number. The main limitations are the permissive matching criterion and the absence of variance information, which I discuss below; neither issue reflects circularity, because the test islands and the pretrained weights provide independent grounding for the empirical comparison.","major_comments":[{"comment":"The F1 score is computed with IoU threshold 0.1 and confidence threshold 0.1 against 50x50-pixel pseudo-boxes. As the paper acknowledges, the low IoU threshold was chosen because the pseudo-boxes are inaccurate, but at IoU 0.1 a predicted box of the same size can be offset by roughly 40 pixels and still be counted as a true positive. The reported F1 therefore mainly measures whether a detection falls in the broad neighborhood of a manual point rather than whether the predicted box localizes the bird. Because the manuscript's central claim is a marked improvement in detection accuracy, the metric definition is load-bearing. I ask for a sensitivity check at stricter IoU thresholds (e.g., 0.3 and 0.5) with precision and recall reported separately, so the reader can see whether the ordering in Table 1 survives when localization is required.","section":"Section 3.2, Table 1"},{"comment":"The ground truth is a single set of point annotations converted to fixed 50x50 boxes, and the paper states that annotators sometimes offset points and that the selected set was chosen by visual inspection for comprehensiveness. This makes the pseudo-boxes a noisy proxy for true bird extents, and inter-observer variability is acknowledged but not quantified. Because all variants are evaluated against the same noisy labels, the relative ordering of methods may be robust, but the absolute F1 values and the statement that 29.25% of detections are false positives are optimistic, or at least unverified. Please quantify annotation uncertainty (e.g., annotator agreement on a subset of tiles) or discuss how label noise affects the main comparisons.","section":"Section 2, Section 3.2"},{"comment":"Each leave-one-island-out fold appears to be run once; no random seeds, repeated runs, confidence intervals, or significance tests are reported. The headline gain from 0.7045 (fine-tuned) to 0.7504 (fine-tuned with stronger augmentations) is an average over eight islands, and island-level differences can be small or, for Tunnel Island with baseline fine-tuning, negative relative to zero-shot. Without variance estimates, the marked-improvement claim is not distinguishable from training stochasticity. Please provide results over multiple seeds with means and standard deviations, or a paired statistical test over the eight islands.","section":"Section 3.2, Table 1"}],"minor_comments":[{"comment":"The text contains the typo 'UA Vs' where 'UAVs' is intended; this appears twice in the introduction.","section":"Section 1"},{"comment":"The 'stronger augmentation' condition is not fully specified: the ranges for brightness, contrast, and HSV adjustments, and the probabilities of the individual transformations, are omitted, which makes this key training configuration hard to reproduce. Please add these details or a pointer to released code.","section":"Section 3.1"},{"comment":"The sentence 'an average of 29.25% of the detections were false positives' is ambiguous: it should state whether this is the mean of per-island false-positive rates or a pooled fraction over all detections.","section":"Section 5"},{"comment":"The caption says the colored boxes correspond to true positives, false positives, and false negatives, but if the figure is viewed in grayscale, the green/red/blue distinction may be lost; consider adding symbols or a separate legend.","section":"Figure 2"},{"comment":"The phrase 'significantly more false positives' uses 'significantly' without a statistical test; 'substantially' would be more accurate unless a test is reported.","section":"Section 4"},{"comment":"The manuscript does not state whether the code, trained weights, or annotations will be made available; an availability statement would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical direction is credible and the paper is within the journal's scope. My recommendation of major revision is driven by the threshold sensitivity and the lack of variance analysis, not by concerns about circularity. I would be satisfied if the authors add a stricter-IoU sensitivity check and multiple-seed results, or a clear justification for why a single run per fold is sufficient for the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about applied conservation ML, but don't expect a methodological breakthrough. The paper does something simple and mostly does it transparently: it takes the existing BirdDetector, fine-tunes it on point-annotated drone imagery of Salvin's albatross, adds SAHI and a stronger augmentation set, and reports per-island F1 under leave-one-island-out cross-validation. The improvement from 0.6576 to 0.7504 average F1 is real in the sense that the table consistently favors the fine-tuned models, and the authors are upfront about variations across islands and about the persistence of false positives on rocky terrain.\n\nThe best part is the honesty: per-island numbers are all shown, thresholds are fixed and not tuned on validation, and the paper explicitly acknowledges that the low IoU threshold exists because the ground truth boxes are pseudo-boxes derived from point annotations. That candor is exactly why the stress-test concern lands but doesn't sink the paper. Yes, an IoU of 0.1 with a fixed 50x50 pseudo-box measures \"did a detection land in the broad neighborhood of a manual point\" rather than tight localization. For a counting task that is actually the relevant operational definition, since you mainly need one detection per bird, not a precise box. And because the same loose threshold is applied to every condition, the relative ordering between zero-shot and fine-tuned is not an artifact of the matching rule. The bigger worry is that there are no repeated runs, no confidence intervals, and no sensitivity analysis at stricter IoU thresholds. With single runs, the 0.09 gain from fine-tuning could in principle be noise, though the consistent direction across eight islands makes that less likely.\n\nThe other soft spots are more minor: evaluation only on tiles that contain annotated birds, so detection behavior on empty or unannotated areas is not characterized; and no public code or data, which limits independent checking. The paper is an empirical evaluation of known techniques, not a new method, but that is fine for the target audience of ecologists and conservation practitioners who need a worked example with a vulnerable species.\n\nMy take: give it a serious referee. Ask for sensitivity analyses at IoU 0.1/0.3/0.5, repeated seeds or error bars, and ideally the annotation and model-release plan. Those are cheap additions that would substantially firm up the central claim. I would not desk-reject it; I'd also not cite it for the method itself, but I might cite it as a recent application example in drone-based seabird monitoring.","headline":"A modest but honest applied transfer-learning study: fine-tuning BirdDetector with stronger augmentations improves F1 on a new albatross dataset, but permissive IoU matching and missing error bars make the exact gain softer than reported.","tokens_in":9223,"tokens_out":1560,"would_cite":true,"duration_ms":18405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a general bird detector with stronger augmentations lifts F1 on Salvin's albatross surveys from 0.6576 to 0.7504.","keywords":["Salvin's albatross","wildlife monitoring","UAV surveys","object detection","BirdDetector","fine-tuning","data augmentation","slicing-aided hyper-inference"],"falsifier":"Re-annotate one held-out island with precise bounding boxes around every visible albatross, run the best fine-tuned model on that island, and recompute F1 at intersection-over-union thresholds from 0.1 to 0.5. If the score collapses as the threshold rises, the reported improvement is largely an artefact of loose box matching rather than real localisation quality; a simpler check is comparing automated counts island by island with independent manual counts on full orthomosaics and seeing whether false positives and false negatives balance.","tokens_in":8208,"feed_emoji":"🐦","tokens_out":7921,"duration_ms":65887,"temperature":0.7,"pith_summary":"This paper asks whether a general-purpose avian detector can be turned into a reliable counter of Salvin's albatrosses on drone imagery from the remote Bounty Islands. Using leave-one-island-out evaluation, it shows that fine-tuning the model on annotated tiles from seven islands raises average F1 from 0.6576 for the zero-shot baseline with slicing inference to 0.7045 with standard fine-tuning, and to 0.7504 when training adds stronger augmentations. The authors argue that this is enough to substantially accelerate manual population counts, while noting that false positives from rocks, penguins, and seals, plus missed birds, remain. The significance is practical: a threatened seabird nesting on almost inaccessible islands can be surveyed from drones with a model that transfers to an unseen island without per-island retraining.","feed_headline":"Albatross-counting AI improves from 0.66 to 0.75 F1 with fine-tuning","feed_subtitle":"A general avian detector, retrained on seven islands, cuts missed birds and false alarms on the eighth island.","key_machinery":"The load-bearing object is BirdDetector, a general avian detector built on a RetinaNet with a ResNet-50 backbone. The authors keep its architecture and default hyperparameters, change test-time inference to Slicing-Aided Hyper-Inference, which cuts images into overlapping 1000 by 1000 pixel patches and merges detections, and replace the original augmentation set with one that adds HSV shifts, random flips, and random crops of 700 to 1200 pixels resized to 1000 by 1000. Ground truth is supplied as 50 by 50 pixel pseudo-bounding boxes centred on manual point annotations from the drone survey. These components work together: the pretrained detector supplies transferable bird features, slicing inference improves small-object recall, and the stronger augmentation set is what pushes the fine-tuned model past the zero-shot baseline.","core_discovery":"The central claim is that fine-tuning with target-domain annotations and stronger data augmentation markedly improves detection accuracy over zero-shot inference, and that the improvement holds across eight held-out islands. In the best configuration, average F1 reaches 0.7504, versus 0.6576 for the zero-shot model with slicing-aided inference and 0.5018 for zero-shot without it. Detections are scored against pseudo-bounding boxes made by centring 50 by 50 pixel squares on manual point annotations, with a deliberately low intersection-over-union threshold of 0.1 to absorb annotation offset. The paper interprets the results as evidence that stronger augmentation simulates variation in flight altitude and lighting, improving generalisation to unseen islands, and that overlapping-tile inference will be preferable for future whole-island counts.","pith_inferences":["Beyond the paper, the loose IoU threshold of 0.1 means F1 measures whether a detection lands near a bird, not how precisely it locates one; a counting application could report count error directly.","Beyond the paper, the same manual point annotations could support a density-map counting baseline, which would test whether bounding-box detection is even necessary for accurate population estimates.","Beyond the paper, the many false positives from penguins and seals suggest that a two-stage pipeline that first finds all birds and then separates species could improve precision more than further augmentation tuning.","Beyond the paper, applying the best model to a colony with different lighting or substrate would directly test the paper's hint that performance depends on visual similarity to the training islands."],"forward_implications":["On a held-out island, the strongest fine-tuned configuration reaches average F1 of 0.7504 versus 0.6576 for zero-shot with SAHI, so fine-tuning transfers across islands without per-island training.","Stronger augmentations usually raise precision at the cost of recall, so the augmentation choice should be tuned to whether a survey prioritises avoiding false alarms or avoiding missed birds.","Slicing-aided inference helps the zero-shot model substantially (0.6576 versus 0.5018), and the paper expects overlapping-tile inference on full orthomosaics to reduce double counting in whole-island population counts.","With averages of 29.25 percent false positives and 19.3 percent missed birds, the model partially automates the count workflow rather than fully replacing it.","Retraining on all eight islands instead of leaving one out is expected to improve performance beyond the cross-validation estimates."],"supporting_citations":[{"why":"Supplies the pretrained general-purpose avian detector, BirdDetector, that all experiments fine-tune from.","marker":"[40]"},{"why":"Provides the DeepForest RetinaNet architecture that BirdDetector is adapted from and the fixed evaluation thresholds used here.","marker":"[39]"},{"why":"Provides the Slicing-Aided Hyper-Inference method used for test-time tiling and merging of detections.","marker":"[2]"},{"why":"Previous drone survey of the Bounty Islands whose imagery and manual point annotations this study reuses.","marker":"[30]"},{"why":"Documents the drone survey data collection that produced the orthomosaic imagery analysed in the study.","marker":"[31]"},{"why":"Earlier albatross counting work that documents ground-truth uncertainty and inter-observer variability, motivating the pseudo-box and IoU choices.","marker":"[6]"}],"fun_headline_variants":["Fine-tuned bird AI boosts albatross detection F1 to 0.75","Retraining general avian model lifts albatross F1 from 0.66 to 0.75","Albatross counting AI improves 14% with fine-tuning on target data","Zero-shot to fine-tuned: albatross detector F1 jumps 0.66 to 0.75","Fine-tuning general bird AI raises albatross count accuracy on new islands"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats manual point annotations, expanded to 50 by 50 pixel boxes and matched at an intersection-over-union threshold of only 0.1, as ground truth; if annotations miss birds, are systematically offset, or the permissive threshold credits detections that are not actually on birds, the reported F1 values overstate detection quality.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned bird AI boosts albatross detection F1 to 0.75","Retraining general avian model lifts albatross F1 from 0.66 to 0.75","Albatross counting AI improves 14% with fine-tuning on target data","Zero-shot to fine-tuned: albatross detector F1 jumps 0.66 to 0.75","Fine-tuning general bird AI raises albatross count accuracy on new islands"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1718,"prompt_tokens":887,"completion_tokens":831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":715}},"tokens_in":503,"tokens_out":831,"duration_ms":7137,"temperature":1.0,"reasoning_tokens":715,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:04:18.219942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate one held-out island with precise bounding boxes around every visible albatross, run the best fine-tuned model on that island, and recompute F1 at intersection-over-union thresholds from 0.1 to 0.5. If the score collapses as the threshold rises, the reported improvement is largely an artefact of loose box matching rather than real localisation quality; a simpler check is comparing automated counts island by island with independent manual counts on full orthomosaics and seeing whether false positives and false negatives balance.","supporting_citations":[{"cited_title":"Weinstein, Lindsey Garner, Vienna R","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained general-purpose avian detector, BirdDetector, that all experiments fine-tune from."},{"cited_title":"Weinstein, Sergio Marconi, Stephanie Bohlman, Alina Zare, and Ethan White","cited_arxiv_id":null,"evidence_quote":"Provides the DeepForest RetinaNet architecture that BirdDetector is adapted from and the fixed evaluation thresholds used here."},{"cited_title":"Slicing aided hyper inference and fine-tuning for small object detection","cited_arxiv_id":null,"evidence_quote":"Provides the Slicing-Aided Hyper-Inference method used for test-time tiling and merging of detections."},{"cited_title":"Sagar, and David R","cited_arxiv_id":null,"evidence_quote":"Previous drone survey of the Bounty Islands whose imagery and manual point annotations this study reuses."},{"cited_title":"Mattern, David M","cited_arxiv_id":null,"evidence_quote":"Documents the drone survey data collection that produced the orthomosaic imagery analysed in the study."},{"cited_title":"Fretwell, Geoffrey French, and Michal Mackiewicz","cited_arxiv_id":null,"evidence_quote":"Earlier albatross counting work that documents ground-truth uncertainty and inter-observer variability, motivating the pseudo-box and IoU choices."}],"review_version":1}