Pith. sign in

REVIEW 4 major objections 6 minor 20 references

DeepSea MOT: A benchmark dataset for multi-object tracking on deep-sea video

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents DeepSea MOT, the first publicly available benchmark for multi-object tracking in deep-sea video, with four annotated ROV sequences and HOTA baselines.

desk verdict A genuinely new and publicly checkable deep-sea MOT benchmark, but the headline tuning gains and model rankings sit on ground truth seeded by the same tuned tracker, so treat the quantitative claims as provisional. read the letter →

arxiv 2509.03499 v1 pith:MBS3RGWG submitted 2025-09-03 cs.CV

Kevin Barnard , Elaine Liu , Kristine Walz , Brian Schlining , Nancy Jacobsen Stout , Lonny Lundsten This is my paper · ORCID
classification cs.CV
keywords multi-objecttrackingdeep-seavideobenchmarkdatasetHOTAByteTrackobjectdetectionROVmidwaterandbenthichabitats
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces DeepSea MOT, a benchmark dataset of four ROV video sequences from midwater and benthic habitats, and claims it is the first publicly available resource for evaluating multi-object tracking in deep-sea footage. The authors argue that existing benchmarks focus on pedestrians, vehicles, or terrestrial animals and do not capture the deep-sea challenges of low visibility, marine snow, transparent organisms, high density, and occlusion. They evaluate three object detection models with ByteTrack and report performance with HOTA, finding that a manually tuned tracker outperforms default settings by roughly 15% on average. The contribution is a reusable evaluation resource plus a documented workflow and notebooks so other groups can build and use similar benchmarks.

What carries the argument

The load-bearing object is the benchmark dataset itself: four 600-frame sequences with manually corrected, ML-seeded ground-truth tracks, spanning habitats and difficulty levels. The evaluation machinery is HOTA (Higher Order Tracking Accuracy), a metric that equally weights detection, localization, and association, computed with TrackEval. ByteTrack, as implemented in Ultralytics, supplies the tracker; its default versus tuned hyperparameters are the experimental variable that produces the paper's main quantitative finding.

What would settle it

Re-annotate a sample of frames from each of the four videos from raw footage without showing annotators the ML-generated tracks, then compare the blind annotations frame-by-frame to the released ground truth; if the blind pass adds, moves, or deletes boxes in ways that correlate with the seeding model's known weaknesses, the reported HOTA scores and the tuned-versus-default gap would change.

Watch

Extended reading notes

Core claim

The central claim is that DeepSea MOT fills a gap in tracking benchmarks by providing human-verified ground truth for deep-sea ROV video. The dataset contains simple and difficult sequences for both midwater and benthic environments, with 8 to 94 tracks per sequence and up to 47.8 boxes per frame. Using HOTA, the authors report that the MBARI 452k model with tuned ByteTrack reaches 76.776% HOTA on the simplest midwater video, while the hardest midwater video yields the lowest scores, down to 43.497% for the FathomNet Megalodon detector. The paper further reports that tuning ByteTrack's association thresholds and buffer settings improved HOTA by about 15% on average, and that additional track

Load-bearing premise

The benchmark's ground-truth annotations are treated as a gold standard even though they were seeded by the same kind of detector being evaluated and then manually corrected, with no reported check for systematic misses that the seeding model may have introduced.

Editorial extensions

If this is right

  • Deep-sea object detectors and trackers can now be compared on a common, public evaluation set instead of ad hoc internal tests.
  • Groups with their own ROV footage can follow the documented workflow to create additional benchmark sequences tailored to their habitats and taxa.
  • HOTA submetrics such as recall, precision, and association accuracy provide structured feedback for improving detection and tracking in deep-sea conditions.
  • Reported scores give a baseline for future models: improving on 76.776% HOTA for the simple midwater sequence and 70.508% for the difficult benthic sequence would demonstrate progress.
  • The large gap between default and tuned tracker settings suggests that domain-specific tracker calibration matters as much as model choice in this environment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ML-seeded, manually corrected annotations carry any systematic bias from the seeding model, all reported HOTA scores and the tuned-versus-default comparison could shift; an independent blind re-annotation study would test this directly.
  • The workflow could be adopted by other deep-sea and marine video archives, potentially creating a family of habitat-specific MOT benchmarks that together cover more taxa and camera geometries than this first set.
  • The deep-sea failure modes named here—transparent bodies, marine snow, camera motion, and dense aggregations—may make this benchmark useful for stress-testing trackers designed for other transparent or low-visibility domains.
  • Because the paper reports that no metric-driven tuning improved on the visually tuned tracker, the data could be used to study how HOTA gradients behave near an apparently well-optimized tracker configuration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DeepSea MOT, a publicly released benchmark for multi-object tracking in deep-sea ROV video footage. The dataset consists of four 600-frame sequences (midwater simple/difficult and benthic simple/difficult) with manually corrected ground-truth tracks. The paper evaluates three object detection models (MBARI 315k, MBARI 452k, FathomNet Megalodon) combined with ByteTrack, comparing default and tuned tracker hyperparameters, and reports HOTA scores. The central claims are that this is the first public deep-sea MOT benchmark, that model rankings and the tuned-versus-default tracker comparison are meaningful, and that the full pipeline (data, code, documentation, notebooks) is publicly available.

Significance. If the resource is sound, DeepSea MOT fills a genuine gap: deep-sea video has no standard MOT benchmark despite unique challenges (transparent organisms, marine snow, camera motion, dense aggregations). The paper ships the dataset on Hugging Face, code on GitHub, and detailed documentation, which makes the primary resource claim externally checkable and lowers the burden of proof for existence. The authors also provide a reproducible workflow for extending the benchmark, which is valuable. However, the quantitative evaluation—the ~15% tuned-versus-default gap and the model rankings in Table II—rests on ground-truth construction that is seeded by the very model and trackers under evaluation, with no reported inter-annotator agreement, repair statistics, or training-data separation analysis. These issues must be resolved before the numeric conclusions can be accepted.

major comments (4)
  1. [Section II (Ground Truth Generation)] The ground-truth construction is the load-bearing component for all quantitative claims. The paper states that each video was first run through MBARI's 452k model and tuned ByteTrack to produce preliminary ML-generated tracks, and that annotators then manually fit localizations or created missing ones. If the annotators retained ML-assigned track IDs or if the manual pass was anchored by the seeded detections, the tuned ByteTrack is partially scored against its own output, and the reported 15% improvement in Figure 3 may be inflated. The paper also mentions in Section III that faint Poeobius meseres are often missed, so the 452k model's blind spots could be encoded in the ground truth. Please provide per-video repair statistics (boxes added, deleted, moved, tracks re-associated, IDs re-assigned), an inter-annotator agreement study, and an explicit statement of how track IDs were created/
  2. [Table II / Figures 2-3] All HOTA values are single point estimates with no variance, confidence intervals, or inter-annotator uncertainty. This matters directly for the model ranking: the BD difference between MBARI 452k (70.202) and MBARI 315k (70.508) is only 0.306 HOTA points, which is likely within annotation noise, yet it is used to state that 315k 'garnered the highest score' on BD. Similarly, the default-vs-tuned ByteTrack comparison in Figure 3 has no error bars. At minimum, report repeated inference runs with different random seeds (if any), or provide a calibration of annotation noise by having at least two annotators label a subset and computing HOTA stability.
  3. [Section II / Section III (Training Data Independence)] The three MBARI detection models were trained on MBARI imagery from the same archive from which the benchmark videos were selected. The paper does not state whether the benchmark sequences, or frames temporally near them, were excluded from model training. If any of the four benchmark videos overlap with training data, the reported HOTA scores are optimistic in a way that varies per model, biasing the model rankings in Table II. Please disclose the temporal and dive identity of the benchmark videos and confirm they were held out from training for all evaluated models.
  4. [Appendix A and Table II] The detection inference parameters differ across models: MBARI models use conf=0.001, while FathomNet Megalodon uses conf=0.15. This is likely intentional because the latter was trained differently, but it means the comparison in Table II is not a controlled comparison of detection architectures; it is a comparison of deployment-ready systems. The text should explicitly acknowledge that the rankings could change if a common inference threshold were used, and ideally report the sensitivity of HOTA to the confidence threshold for at least one model.
minor comments (6)
  1. [Abstract / Section II] The abstract says 'human-generated test data,' but the ground truth is ML-seeded and then manually corrected. Recommend changing to 'expert-corrected, ML-seeded ground truth' to avoid ambiguity.
  2. [Throughout] Several formatting issues: 'I NTRODUCTION' in Section I, 'V ARS' should be 'VARS', and 'T rackEval' should be 'TrackEval'.
  3. [References] References [9] and [10] are the same URL and should be merged or clearly differentiated. Also, some references lack publisher information.
  4. [Appendix A] The appendix says 'evaluated in section II' but the experiments are in Section III. Please correct the cross-reference.
  5. [Section II] The term '452k model' is used without definition. Even if the model name is conventional internally, a one-line description (e.g., number of training images or resolution) would help external readers.
  6. [Figures 2-3] The bar charts show HOTA scores but do not include the submetrics (DetA, AssA, LocA) that are mentioned in the text. Including them or making the notebook visible would strengthen reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Ground-truth association is seeded by the tuned ByteTrack being evaluated, so the reported ~15% tuning gain is partially circular by construction.

  1. fitted input called prediction [Section II (Materials and Methods), ground-truth generation; Section III (Results), Figure 3 and Table II]
    "each video was first run through MBARI's 452k model and tuned ByteTrack tracker to produce preliminary ML-generated tracks. The results and corresponding video frames were then imported to RectLabel, an image annotation tool, to manually fit localizations to the object or create them when objects were not detected by the model. After all 600 frames were reviewed for each video sequence, the resulting ground truth files were then considered a gold-standard benchmark, available for comparison to the model output files for evaluation."

    TrackEval's HOTA metric balances detection, localization, and association; the association component depends on ground-truth track IDs. The paper states that these IDs were initialized by 'MBARI's 452k model and tuned ByteTrack tracker' and describes human correction only as 'manually fit localizations' (bounding boxes), not as re-association of track IDs. Therefore the tuned ByteTrack is scored against a ground truth whose object trajectories it generated, while the default ByteTrack must recover the same trajectories without the tuned thresholds/buffer. The headline result that the default tracker is 'approximately 15% less' (Section III, Figure 3) is thus partly forced by the ground-truth construction itself, at least for the association submetric, rather than being an independent measu

full rationale

The paper is primarily a benchmark resource paper: it constructs four deep-sea video sequences with human-reviewed bounding-box localizations, computes HOTA via the standard TrackEval implementation, and makes code, data, and documentation public. There is no load-bearing self-citation chain or imported uniqueness theorem; the first-benchmark claim and the benchmark's usability are externally checkable. However, the central quantitative comparison in Section III is not fully independent. The ground-truth track IDs originate from the tuned ByteTrack tracker whose tuned-versus-default performance is reported as a key finding. Because human correction is described only as fitting/creating localizations and no manual track-ID re-association or repair statistics are reported, the association component of the tuned tracker's HOTA is a self-match against a target it helped define. This partially manufactures the reported 15% tuning advantage and also affects all Table II HOTA scores through AssA. The detection/localization portion of the ground truth was human-reviewed, so the circularity is partial rather than total; the resource itself remains valuable. Score 6 reflects one load-bearing evaluation claim that reduces by construction. The separate concern that tracker hyperparameters were tuned by visual analysis on unspecified MBARI imagery is noted in the paper but not sufficiently specified to count as a demonstrated circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The reported scores depend on two hand-tuned parameter sets (tracker and inference thresholds), on the assumption that ML-seeded manual labels are unbiased, on HOTA/TrackEval, and on the transferability of tuning to the benchmark videos. The dataset itself is the only new entity and it is publicly checkable; no theoretical postulates are introduced.

free parameters (2)
  • ByteTrack tuned hyperparameters = track_high_thresh=0.01, track_low_thresh=0.005, new_track_thresh=0.001, track_buffer=300, match_thresh=0.80, fuse_score=
    Set by author K. Walz through iterative visual tuning on MBARI imagery (Section II, Appendix B); the tuning footage is not specified. These values drive the headline ~15% HOTA gain of tuned over default ByteTrack (Figure 3), so the central comparison rests on hand-chosen numbers.
  • Detection inference thresholds (per model) = conf: 0.001 (315k, 452k), 0.15 (FNMD); iou: 0.2 (315k, 452k), 0.6 (FNMD); imgsz 1280; augment=True; agnostic_nms=True
    Called 'previously optimized for accuracy with MBARI's typical imagery' (Section II, Appendix A); no search procedure or validation data are given, and the differing thresholds across models complicate the cross-model HOTA comparison in Table II.
assumptions (4)
  • domain assumption The ML-seeded, manually corrected annotations are a gold-standard ground truth with no systematic bias.
    Section II: tracks from the 452k model and tuned ByteTrack were hand-corrected in RectLabel and 'considered a gold-standard benchmark'; no inter-annotator agreement or repair statistics are reported, and the annotators belong to the institution that trained the evaluated models.
  • domain assumption The four sequences are representative of midwater and benthic deep-sea habitats, with difficulty captured by animal density.
    Section II and Table I: difficulty is 'defined largely by the number of animals present'; the paper's conclusion that content beats habitat is inferred from these two pairs of videos (n=4).
  • standard math HOTA implemented by TrackEval is a valid and correctly computed metric for this domain.
    Section II cites [12], [13]; the metric definition is taken as given and not re-derived, standard practice for benchmark papers.
  • domain assumption The hand-tuned inference and tracking parameters transfer to the benchmark videos without test-set leakage.
    Appendices A and B: parameters were tuned 'previously' on unspecified imagery; the paper states metrics were not used for tuning (Section III) but does not demonstrate that the benchmark videos were excluded from the visual tuning.
invented entities (1)
  • DeepSea MOT benchmark dataset (4 ROV video sequences, ground-truth files) independent evidence
    purpose: Public evaluation standard for deep-sea multi-object tracking
    A constructed public asset rather than a postulated phenomenon; it is downloadable from Hugging Face and checkable end-to-end, so its existence carries independent evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepSea MOT: A benchmark dataset for multi-object tracking on deep-sea video." pith.science (2026). https://pith.science/paper/MBS3RGWG

@misc{pith2026250903499,
  author       = {Pith},
  title        = {Pith review of: DeepSea MOT: A benchmark dataset for multi-object tracking on deep-sea video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MBS3RGWG}},
  note         = {Machine review of arXiv:2509.03499}
}
read the original abstract

Benchmarking multi-object tracking and object detection model performance is an essential step in machine learning model development, as it allows researchers to evaluate model detection and tracker performance on human-generated 'test' data, facilitating consistent comparisons between models and trackers and aiding performance optimization. In this study, a novel benchmark video dataset was developed and used to assess the performance of several Monterey Bay Aquarium Research Institute object detection models and a FathomNet single-class object detection model together with several trackers. The dataset consists of four video sequences representing midwater and benthic deep-sea habitats. Performance was evaluated using Higher Order Tracking Accuracy, a metric that balances detection, localization, and association accuracy. To the best of our knowledge, this is the first publicly available benchmark for multi-object tracking in deep-sea video footage. We provide the benchmark data, a clearly documented workflow for generating additional benchmark videos, as well as example Python notebooks for computing metrics.

Figures

Figures reproduced from arXiv: 2509.03499 by the authors.

Figure 1
Figure 1. Example frames from benthic and midwater benchmark videos. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A comparison of HOTA scores for the three object [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A comparison of HOTA scores for four benchmark [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

  1. [1]

    MBARI’s video annotation and reference system,

    B. Schlining and N. Stout, “MBARI’s video annotation and reference system,” in MTS/IEEE Oceans Conference , 2006

    work page 2006

  2. [2]

    Visual tracking of deep- water animals using machine learning-controlled robotic underwater vehicles,

    K. Katija, P. Roberts, J. Daniels, A. Lapides, K. Barnard, M. Risi, B. Ranaan, B. Woodward, and J. Takahashi, “Visual tracking of deep- water animals using machine learning-controlled robotic underwater vehicles,” in IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2021, pp. 859–868

    work page 2021

  3. [3]

    MOTChal- lenge 2015: Towards a benchmark for multi-target tracking,

    L. Leal-Taix ´e, A. Milan, I. Reid, S. Roth, and K. Schindler, “MOTChal- lenge 2015: Towards a benchmark for multi-target tracking,” 2015, https://arxiv.org/abs/1504.01942. 5

    arXiv 2015

  4. [4]

    AnimalTrack: A benchmark for multi-animal tracking in the wild,

    L. Zhang, J. Gao, Z. Xiao, and H. Fan, “AnimalTrack: A benchmark for multi-animal tracking in the wild,” International Journal of Computer Vision, vol. 131, pp. 496–513, 2023

    work page 2023

  5. [5]

    Object tracking benchmark,

    Y . Wu, J. Lim, and M.-H. Yang, “Object tracking benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 37, no. 9, pp. 1834–1848, 2015

    work page 2015

  6. [6]

    Transparent object tracking benchmark,

    H. Fan, H. A. Miththanthaya, H. Harshit, S. R. Rajan, X. Liu, Z. Zou, Y . Lin, and H. Ling, “Transparent object tracking benchmark,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , 2021, pp. 10 714–10 723

    work page 2021

  7. [7]

    Light in the ocean’s midwaters,

    B. Robison, “Light in the ocean’s midwaters,” Scientific American, vol. 273, pp. 49–56, 1995

    work page 1995

  8. [8]

    Gage and P

    J. Gage and P. Tyler, Deep-Sea Biology: A Natural History of Organisms at the Deep-Sea Floor . Cambridge University Press, 1991

    work page 1991

Show all 20 references
  1. [9]

    Ultralytics YOLOv8,

    G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics

    2023

  2. [10]

    Ultralytics YOLOv8,

    ——, “Ultralytics YOLOv8,” https://github.com/ultralytics/ultralytics, 2023

    2023

  3. [11]

    ByteTrack,

    Y . Zhang, P. Sun, Y . Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “ByteTrack,” inEuropean Conference on Computer Vision (ECCV), 2022

    2022

  4. [12]

    HOTA: A higher order metric for evaluating MOT,

    J. Luiten, A. Osep, P. Dendorfer et al., “HOTA: A higher order metric for evaluating MOT,” International Journal of Computer Vision , vol. 129, pp. 548–578, 2021

    2021

  5. [13]

    TrackEval,

    J. Luiten and A. Hoffhues, “TrackEval,” https://github.com/ JonathonLuiten/TrackEval, 2020

    2020

  6. [14]

    FathomNet Megalodon object detector,

    L. Lundsten and L. Chrobak, “FathomNet Megalodon object detector,” https://huggingface.co/FathomNet/megalodon, 2024

    2024

  7. [15]

    BoT-SORT: Robust associa- tions multi-pedestrian tracking,

    N. Aharon, R. Orfaig, and B. Bobrovsky, “BoT-SORT: Robust associa- tions multi-pedestrian tracking,” arXiv preprint arXiv:2206.14651, 2022

    2022 arXiv

  8. [16]

    BoxMOT: pluggable SOTA tracking modules for object detection, segmentation and pose estimation models,

    M. Brostr ¨om, “BoxMOT: pluggable SOTA tracking modules for object detection, segmentation and pose estimation models,” Zenodo, 2023

    2023

  9. [17]

    BoostTrack,

    V . Stanojevic and B. Todorovi, “BoostTrack,” Machine Vision and Applications, 2024

    2024

  10. [18]

    StrongSORT,

    Y . Du, Z. Zhao, Y . Song, Y . Zhao, F. Su, T. Gong, and H. Meng, “StrongSORT,” IEEE Transactions on Multimedia , vol. 25, pp. 8725– 8737, 2023

    2023

  11. [19]

    Observation- centric SORT,

    J. Cao, J. Pang, X. Weng, R. Khirodka, and K. Kitani, “Observation- centric SORT,” in Proceedings of CVPR , 2023

    2023

  12. [20]

    Deep OC-SORT,

    G. Maggiolino, A. Ahmad, J. Cao, and K. Kitani, “Deep OC-SORT,” 2023, arXiv:2302.11813. APPENDIX A DETECTION PARAMETERS The following table gives the non-default detection param- eters used for each of the three models evaluated in section II per the implementation provided by...

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.