Pith. sign in

REVIEW 4 major objections 6 minor 32 references

No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mixed annotation strategy beats blind foundation-model labeling for animal tracking.

desk verdict Useful, honest evaluation of foundation-model-assisted annotation for rodent tracking, but the headline gain is confounded by fine-tuning versus zero-shot comparison and needs an ablation. read the letter →

arxiv 2502.03907 v1 pith:3DZUP3DR submitted 2025-02-06 cs.CV

Emil Mededovic , Valdy Laurentius , Yuli Wu , Marcin Kopaczka , Zhu Chen , Mareike Schulz , René Tolba , Johannes Stegmaier This is my paper · ORCID
classification cs.CV
keywords foundationmodelsannotationanimaltrackingSAM-QAsegmentanythingqualityassessmentIDF1semi-automatic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether foundation-model-generated annotations can replace or supplement manual labels when training an animal tracker, and it answers with a caution: blindly feeding SAM2 video masks into training hurts downstream tracking. The authors propose SAM-QA, a semi-automatic pipeline in which a lightweight, fine-tuned SAM produces masks from bounding-box prompts, a spatio-temporal consistency check rejects poor masks, and SAM2 recovers when checks fail. On rodent videos, a tracker trained on SAM-QA labels reaches 80.8 IDF1 on rats, versus 65.6 for a tracker trained on zero-shot SAM2 video labels, while fully manual labels reach 85.0. The paper concludes that thoughtful mixing of automated and manual annotations is beneficial but still falls short of manual annotation, especially for the harder mouse dataset.

What carries the argument

The load-bearing mechanism is the SAM-QA annotation loop. A distilled SAM with a TinyViT encoder, fine-tuned on the target animal videos, generates binary masks from bounding-box prompts; before a mask is accepted, a validation step checks spatio-temporal consistency against the previous frame using an IoU overlap threshold (β = 0.9) and an area-ratio window (α = 0.1); if the check fails, SAM2 with equidistant grid-sampled point prompts attempts automatic recovery, and only if that fails does the user re-initialize manually. This loop converts a small number of manual bounding boxes into a stream of verified masks, and it is the consistency check plus targeted recovery that the paper credits for the downstream tracking gain.

What would settle it

Run an ablation in which SAM2 video is fine-tuned on the same rat and mouse training frames and evaluated with the same manual prompt intervals and the same downstream ByteTrack/YOLOv8 setup. If fine-tuned SAM2 video reaches or exceeds 80.8 IDF1 on rats, the proposed consistency check and iterative prompting are not the source of the improvement.

Watch

Extended reading notes

Core claim

The central claim is that a careful combination of automated annotations and manually annotated data is a worthwhile strategy for training animal trackers, and that blind use of a foundation model for label generation is counterproductive. Concretely, the paper reports that their SAM-QA pipeline, which uses a distilled and fine-tuned SAM with iterative prompting and a spatio-temporal consistency check, produces training labels that yield an IDF1 of 80.8 for rats and 61.1 for mice, compared with 65.6 and 34.2 for zero-shot SAM2 video. Manual annotation remains the best label source, at 85.0 IDF1 for rats and 76.6 for mice. The authors interpret these results as evidence that foundation-model labels need quality control and targeted human intervention rather than wholesale adoption.

Load-bearing premise

The load-bearing premise is that SAM-QA's better tracking score comes from its iterative prompting and consistency check, rather than from being trained on the target videos while the comparison method is used as-is; the paper does not run an experiment that separates these two factors.

Editorial extensions

If this is right

  • Tracking models trained on SAM-QA labels outperform those trained on zero-shot SAM2 video labels by 15.2 IDF1 on rats and by 26.9 on mice, with a comparable number of manual interventions.
  • Increasing the manual prompt interval for SAM2 video degrades rat tracking sharply, from 65.6 IDF1 at interval 15 to 17.3 at interval 750, so prompt frequency is a key cost-quality knob.
  • SAM-QA still trails manual annotation by 4.2 IDF1 on rats and 15.5 on mice, so automated label generation is not yet a full replacement for human annotation.
  • On the harder mouse dataset, performance is suboptimal across all automated methods, consistent with the paper's warning that small amounts of noisy labels can break a tracking model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate whether the gain comes from fine-tuning SAM on target data or from the consistency check and recovery loop; a reader should not attribute the full 15.2-point improvement to the quality assessment alone.
  • The same 'no free lunch' logic likely applies to other foundation-model labeling tasks: masks that look acceptable frame-by-frame can still inject tracking-breaking inconsistency, so any automated labeling pipeline for a downstream spatio-temporal task needs a temporal consistency gate.
  • A natural extension would be to fine-tune SAM2 video on the same training frames and prompt intervals used for SAM-QA; matching or beating 80.8 IDF1 would indicate that domain fine-tuning, not the iterative prompt mechanism, is the decisive factor.
  • Given the mouse results, fully manual annotation may remain necessary for low-contrast, multi-object scenes until foundation models improve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SAM-QA, an annotation pipeline for rodent tracking that combines a distilled and fine-tuned SAM with iterative bounding-box prompting and a spatio-temporal consistency check, and reports downstream ByteTrack tracking results. On a rat dataset and a mouse dataset, SAM-QA reaches 80.8 IDF1 on rats versus 65.6 for zero-shot SAM-2 video, while manual annotation reaches 85.0. The paper also benchmarks classical segmentation-plus-watershed methods, Grounding DINO, and SAM-2 video at different prompt intervals. The central claim is that a thoughtful combination of automated and manual annotation is a valuable strategy.

Significance. If the headline comparison were clean, the paper would make a useful practical contribution to semi-automatic annotation for animal tracking, where small label errors can break long-horizon trackers. The manuscript ships a concrete tool and pipeline description, evaluates several baselines, and measures downstream tracking with standard metrics, which is creditworthy. However, the central claim is not currently isolated: the proposed method differs from the SAM-2 video baseline in two independent ways (fine-tuning on the target domain versus zero-shot use, and the proposed iterative prompting/consistency-check protocol), and no ablation separates them. The paper also lacks error bars and a quantitative account of annotation effort, so the strength of the practical recommendation is not fully established.

major comments (4)
  1. [Abstract and §3, Table 2] The headline comparison (80.8 vs 65.6 IDF1) does not identify the source of the improvement. SAM-QA is labeled 'Distilled & Finetuned' while SAM-2V is used zero-shot, so the gap could in principle be entirely due to fine-tuning on the target domain rather than the iterative prompting and spatio-temporal consistency check. The text even acknowledges in §3 that 'this performance boost benefits significantly from fine-tuning and targeted intervention,' but no experiment removes fine-tuning from SAM-QA or adds it to SAM-2V. Please add ablations that vary the segmentation model and the quality-assessment pipeline independently, for example: fine-tuned SAM-2V without the consistency check, SAM-QA without fine-tuning, and SAM-2V with the same iterative prompting protocol. Without such ablations the paper's central claim, as stated in the abstract, is not established by the reported experiments.
  2. [§2, spatio-temporal consistency check] The thresholds alpha=0.1 and beta=0.9 are stated to be 'empirically selected,' but the manuscript does not say on which data they were selected, whether a validation split was used, or how sensitive the downstream IDF1 is to these choices. If the same datasets feed both parameter selection and the final Table 2 numbers, the reported 80.8 IDF1 may partially reflect tuning to the evaluation sequences. Please report the selection procedure and a small sensitivity analysis (e.g., alpha in {0.05, 0.1, 0.2}, beta in {0.8, 0.9, 0.95}) on a held-out split.
  3. [§3, Table 2, and Fig. 3] All reported results are single runs without variance estimates. ByteTrack parameters, YOLOv8 training, and the data augmentation pipeline are stochastic, and Table 2 reports only one value per method/dataset. Some differences are small (e.g., manual 85.0 vs SAM-QA 80.8 on rats, or DeepLabv3 60.2 vs SAM-2V 65.6), so it is not possible to assess whether the differences are meaningful. Please provide multiple seeds with mean and standard deviation, or at least report the number of runs and confirm that the observed gaps reproduce.
  4. [§3, 'comparable number of interventions'] The claim that SAM-QA needs 'a comparable number of interventions' to the second-best SAM-2V approach is never quantified. Since the paper's thesis is about streamlining annotation, the number of manual prompts, the number of automatic recovery attempts, and the total annotation time are load-bearing quantities. Please report these numbers per dataset and per method, including the SAM-2V prompt intervals used, so that the reader can judge the human effort required to obtain the 80.8/61.1 IDF1 results.
minor comments (6)
  1. [Title] The title contains an unintended space: 'ANNOTA TION' should be 'ANNOTATION'.
  2. [Fig. 1] The figure caption and the visual text run together ('InitializationIteration' and similar); please separate the labels so that the flow from initialization to iteration to validation is readable.
  3. [Fig. 3] The x-axis labels in Fig. 3 are difficult to read and appear to run together (e.g., '51530150300450600750'); please format the tick labels as '5, 15, 30, 150, 300, 450, 600, 750'.
  4. [§2, Segmentation & Watershed] The training details for the segmentation models (U-Net, DeepLabv3, DINOv2) are incomplete: no number of epochs, optimizer, or input resolution is given, while such details are provided for YOLOv8. Please add them or state that they follow the same protocol.
  5. [§3, Grounding DINO] The sentence 'For a fair comparison, we conducted an analysis where we combined manual and generated annotations across traditional methods and Grounding DINO' is misleading because the comparison in that paragraph is between pure manual, pure generated, and a specific mixed protocol, not a fairness check across methods. Please rephrase to avoid implying the mixed protocol is a baseline for fairness.
  6. [References] Some references are incomplete (e.g., [5] is a preprint from 2023, [8] is a preprint from 2024); if the venue requires published versions, please update them where available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is an empirical benchmarking study; downstream tracking metrics are measured, not derived from fitted inputs.

full rationale

The paper makes no formal derivation claim. Its central result is an empirical comparison of annotation strategies: SAM-QA-generated labels yield an IDF1 of 80.8 on rats versus 65.6 for zero-shot SAM2 video. The downstream IDF1 and HOTA scores are computed by training ByteTrack/YOLOv8 on generated labels and evaluating on held-out sequences; no quantity in Table 2 is obtained by solving for the claim itself. The SAM-QA model is fine-tuned on the target domain, but that is an experimental design confound (the baseline is zero-shot), not a circular step: the paper does not define annotation quality in terms of the final IDF1, nor does it fit a parameter and then present that fitted value as a prediction. The one sentence acknowledging that 'this performance boost benefits significantly from fine-tuning and targeted intervention' (Section 3) is an explicit caveat, not a circularity. No load-bearing self-citations occur: references to SAM [3], SAM2 [8], ByteTrack [20], and similar work are to independent external systems. No uniqueness theorem is imported from the authors' prior work, and no known result is renamed. The potential weakness that the headline gain is not causally isolated from fine-tuning is a validity/attribution concern, which belongs in correctness risk, not circularity analysis. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim does not introduce new physical entities. It relies on several hand-tuned thresholds (alpha, beta, ByteTrack parameters) and domain assumptions about the recording setup and known animal count. These are the main extra ingredients the reader is asked to accept beyond standard methods.

free parameters (4)
  • alpha (consistency check size threshold) = 0.1
    Empirically selected; controls the allowed area change between consecutive masks, directly affecting which masks pass validation.
  • beta (consistency check overlap threshold) = 0.9
    Empirically selected; sets the maximum IoU between masks of different animals in the same frame to flag potential merging.
  • ByteTrack association parameters = track_high_thresh=0.5, track_low_thresh=0.1, match_thresh=0.9, new_track_thresh=0.9, track_buffer=120
    Tuned to the specific rodent tracking problem, as stated in Section 2, affecting the downstream tracking metric.
  • KDE density threshold percentile = 20th percentile
    A fixed design choice in Algorithm 1 for outlier removal in masks, not justified by data or theory.
assumptions (3)
  • domain assumption The number of animals in each frame is known a priori.
    Used for watershed seed selection and Grounding DINO box selection (Section 2). This is a strong assumption that limits generalizability to unconstrained videos.
  • domain assumption The camera is static and the background is unchanging.
    Invoked when justifying the distillation of SAM to a lightweight model for a narrow application with static background (Section 2, SAM-QA).
  • domain assumption IoU-based association and the size/overlap criteria are sufficient to detect incorrect segmentation masks.
    The spatio-temporal consistency check relies on these heuristics, which are not validated independently of the final tracking score.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking." pith.science (2026). https://pith.science/paper/3DZUP3DR

@misc{pith2026250203907,
  author       = {Pith},
  title        = {Pith review of: No Free Lunch in Annotation either: An objective evaluation of foundation models for streamlining annotation in animal tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DZUP3DR}},
  note         = {Machine review of arXiv:2502.03907}
}
read the original abstract

We analyze the capabilities of foundation models addressing the tedious task of generating annotations for animal tracking. Annotating a large amount of data is vital and can be a make-or-break factor for the robustness of a tracking model. Robustness is particularly crucial in animal tracking, as accurate tracking over long time horizons is essential for capturing the behavior of animals. However, generating additional annotations using foundation models can be counterproductive, as the quality of the annotations is just as important. Poorly annotated data can introduce noise and inaccuracies, ultimately compromising the performance and accuracy of the trained model. Over-reliance on automated annotations without ensuring precision can lead to diminished results, making careful oversight and quality control essential in the annotation process. Ultimately, we demonstrate that a thoughtful combination of automated annotations and manually annotated data is a valuable strategy, yielding an IDF1 score of 80.8 against blind usage of SAM2 video with an IDF1 score of 65.6.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 28 canonical work pages

  1. [1]

    To build these robust track- ing models, a significant amount of annotated data is required

    INTRODUCTION Tracking systems that measure animal activity are crucial for assessing stress and severity indicators, providing invaluable insights into animal welfare [1,2]. To build these robust track- ing models, a significant amount of annotated data is required. This demand underscores the need for an efficient, stream- lined annotation process to sup...

    work page

  2. [2]

    METHODS In the following, we provide a detailed overview of the meth- ods for semi-automatic data annotation applied and analyzed in our study, along with an in-depth description of the dataset and the tracker used. Dataset. We have two datasets: a rat dataset with two rats consistently, and a mouse dataset featuring four mice. In Ta- ble 1, the available...

    work page Pith review arXiv 2025

  3. [3]

    Overlap Condition: For masks Mi and Mj in the same frame, the criterion is not met if IoU(Mi, Mj) > β, where β is a user-defined threshold

    work page

  4. [4]

    The criterion is not met if AX /∈ [(1 − α)AY , (1 +α)AY ]

    Size Condition: Let AY be the area of the previous mask (manual prompt) and AX the area of the current mask. The criterion is not met if AX /∈ [(1 − α)AY , (1 +α)AY ]. To pass the spatio-temporal consistency check, all conditions must be satisfied. If this check fails and re-initialization has not yet been performed, the method initiates automatic recov- ...

    work page

  5. [5]

    Per- Clustering (optional) (a) Original (b) Logits (d) Semantic masks (c) Seed regions (e) Watershed Fig

    RESULTS In Table 2, we present the downstream tracking results using newly generated training labels across various methods. Per- Clustering (optional) (a) Original (b) Logits (d) Semantic masks (c) Seed regions (e) Watershed Fig. 2: Illustration of the segmentation process using the wa- tershed method: First, the image is passed through the seg- mentatio...

    work page

  6. [6]

    ACKNOWLEDGMENTS This work was funded by the German Research Foundation DFG with the grants STE2802/4-1 (EM) and STE2802/5-1 (ZC)

    work page

  7. [7]

    Inconsistent bounding boxes can lead to suboptimal model performance due to noise

    CONCLUSION We demonstrate the need for caution when applying founda- tion models for label generation in tracking tasks, given the precision required to maintain track consistency. Inconsistent bounding boxes can lead to suboptimal model performance due to noise. We propose an iterative approach that integrates a lightweight, fine-tuned Segment Anything M...

    work page

  8. [8]

    In SAM-QA, prompts are not set at fixed intervals but are triggered by conflict occurrences, enhancing accuracy

    demonstrates strong and promising performance in zero- shot settings, only underperforming compared to SAM-QA, which uses a fine-tuned SAM and robust quality assessment. In SAM-QA, prompts are not set at fixed intervals but are triggered by conflict occurrences, enhancing accuracy. Since SAM-2V [8] is not fine-tuned, ambiguities arise, potentially impacti...

    work page

Show all 32 references
  1. [9]

    The recordings were sourced from previous studies, ensuring no additional impact on the animals during this work

    COMPLIANCE WITH ETHICAL STANDARDS Ethical approval was not required as the videos were obtained without direct interaction or handling of the animals. The recordings were sourced from previous studies, ensuring no additional impact on the animals during this work

  2. [10]

    Animal Welfare: Severity Assessment in Experimen- tal Research,

    Andre Bleich, Brigitte V ollmar, and Ren ´e H. Tolba, “Animal Welfare: Severity Assessment in Experimen- tal Research,” Eur . Surg. Res., vol. 64, no. 1, pp. 5–6, 2023

    2023

  3. [11]

    Multi-animal pose estimation, iden- tification and tracking with DeepLabCut,

    Jessy Lauer et al., “Multi-animal pose estimation, iden- tification and tracking with DeepLabCut,” Nat. Meth- ods, vol. 19, no. 4, pp. 496–504, 2022

    2022

  4. [12]

    Segment anything,

    Alexander Kirillov et al., “Segment anything,” in ICCV, 2023, pp. 4015–4026

    2023

  5. [13]

    Segment anything for mi- croscopy,

    Anwai Archit et al., “Segment anything for mi- croscopy,” bioRxiv, 2023

    2023

  6. [14]

    Segment anything meets point tracking,

    Frano Raji ˇc, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu, “Segment anything meets point tracking,” arXiv:2307.01197, 2023

    2023 arXiv

  7. [15]

    CoTracker: It is better to track together,

    Nikita Karaev, Ignacio Rocco, Benjamin Graham, Na- talia Neverova, Andrea Vedaldi, and Christian Rup- precht, “CoTracker: It is better to track together,” in ECCV, 2024

    2024

  8. [16]

    PointOdyssey: A large-scale synthetic dataset for long-term point track- ing,

    Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas, “PointOdyssey: A large-scale synthetic dataset for long-term point track- ing,” in ICCV, 2023

    2023

  9. [17]

    SAM 2: Segment anything in im- ages and videos,

    Nikhila Ravi et al., “SAM 2: Segment anything in im- ages and videos,” arXiv:2408.00714, 2024

    2024 arXiv

  10. [18]

    TinyViT: Fast pretraining distillation for small vision transformers,

    Kan Wu et al., “TinyViT: Fast pretraining distillation for small vision transformers,” in ECCV, 2022, pp. 68–85

    2022

  11. [19]

    Generalised dice overlap as a deep learning loss function for highly un- balanced segmentations,

    Carole H Sudre, Wenqi Li, Tom Vercauteren, S ´ebastien Ourselin, and M. Jorge Cardoso, “Generalised dice overlap as a deep learning loss function for highly un- balanced segmentations,” in MICCAI Workshop, 2017, pp. 240–248

    2017

  12. [20]

    Bernard W Silverman, Density estimation for statistics and data analysis, Routledge, 2018

    2018

  13. [21]

    David W Scott, Multivariate density estimation: theory, practice, and visualization, John Wiley & Sons, 2015

    2015

  14. [22]

    U-Net: Convolutional networks for biomedical image segmentation,

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in MICCAI, 2015, pp. 234–241

    2015

  15. [23]

    Rethinking atrous convolution for semantic image segmentation,

    Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam, “Rethinking atrous convolution for semantic image segmentation,” arXiv:1706.05587, 2017

    2017 arXiv

  16. [24]

    DINOv2: Learning robust visual features without supervision,

    Maxime Oquab et al., “DINOv2: Learning robust visual features without supervision,” TMLR, 2024

    2024

  17. [25]

    Use of watersheds in contour detec- tion,

    Serge Beucher, “Use of watersheds in contour detec- tion,” in Proc. Int. Workshop on Image Processing , 1979, pp. 17–21

    1979

  18. [26]

    Least squares quantization in PCM,

    Stuart Lloyd, “Least squares quantization in PCM,” IEEE Trans. Inf. Theory , vol. 28, no. 2, pp. 129–137, 1982

    1982

  19. [27]

    Maximum likelihood from incomplete data via the EM algorithm,

    Arthur P Dempster, Nan M Laird, and Donald B Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. R. Stat. Soc. Ser . B-Stat. Methodol. , vol. 39, no. 1, pp. 1–22, 1977

    1977

  20. [28]

    Grounding DINO: Marrying DINO with grounded pre-training for open-set object detec- tion,

    Shilong Liu et al., “Grounding DINO: Marrying DINO with grounded pre-training for open-set object detec- tion,” in ECCV, 2024

    2024

  21. [29]

    ByteTrack: Multi-object tracking by associating every detection box,

    Yifu Zhang et al., “ByteTrack: Multi-object tracking by associating every detection box,” in ECCV, 2022, pp. 1–21

    2022

  22. [30]

    Ultra- lytics yolov8,

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu, “Ultra- lytics yolov8,” 2023

    2023

  23. [31]

    Performance measures and a data set for multi-target, multi-camera tracking,

    Ergys Ristani, Francesco Solera, Roger Zou, Rita Cuc- chiara, and Carlo Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in ECCV, 2016, pp. 17–35

    2016

  24. [32]

    HOTA: A higher order metric for evaluating multi-object tracking,

    Jonathon Luiten et al., “HOTA: A higher order metric for evaluating multi-object tracking,” Int. J. Comput. Vis, pp. 1–31, 2020

    2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.