Counting Fish with Temporal Representations of Sonar Video

Grant Van Horn; Justin Kay; Kai Van Brunt; Pietro Perona; Sara Beery; Timm Haucke

REVIEW 4 major objections 5 minor 20 references

Counting Fish with Temporal Representations of Sonar Video

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single compressed sonar image per 200 video frames predicts salmon counts with 23% error, replacing per-frame detection and tracking.

desk verdict Direct count regression from echograms is a genuinely new and plausible feasibility result for low-compute salmon counting, but the headline 23% error is measured on a tiny validation set used for model selection and deserves scrutiny. read the letter →

arxiv 2502.05129 v1 pith:VV25RKWJ submitted 2025-02-07 cs.CV

Kai Van Brunt , Justin Kay , Timm Haucke , Pietro Perona , Grant Van Horn , Sara Beery This is my paper · ORCID

classification cs.CV

keywords sonarfishcountingechogramsalmonescapementtemporalvideorepresentationweaklysupervisedlearningimageaugmentationResNet-18rivermonitoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that counting migrating salmon can be done by compressing hundreds of sonar video frames into a single temporal image, called an echogram, and reading the fish count directly from that image with a small neural network. This replaces frame-by-frame object detection and tracking, which requires heavy compute that remote river sites often lack. The authors report a count error of 23 percent on in-distribution validation data and 30.7 percent on an out-of-distribution test set from the Kenai River, and they show that mixing weak labels generated by a detector-tracker with human labels improves accuracy. If the approach holds, salmon escapement monitoring could run on consumer laptops with one forward pass per 200 frames.

What carries the argument

The echogram is the central object: it compresses a multi-beam sonar clip into a single image whose columns are per-frame maximum-intensity profiles over range, with a second channel storing the lateral beam position of that maximum. The model is a ResNet-18 that outputs two non-negative counts, one for upstream and one for downstream fish, for the window. The weakly-supervised training protocol expands the labeled set to 33,437 weak-label images generated by an existing detector-tracker, and the domain-specific augmentations include vertical flipping, superposition of echograms, and a 'realistic horizontal flip' that reflects the image and inverts the lateral-position channel so fish motion direction is preserved.

What would settle it

Measure end-to-end processing time of echogram generation plus ResNet-18 inference per 200-frame window on a typical field laptop and compare it with a detector-tracker on the same hardware; if preprocessing dominates and the total time is comparable to frame-wise detection, the central practical claim fails.

Watch

Extended reading notes

Core claim

The central claim is that upstream and downstream fish counts within a 200-frame time window can be predicted directly from an echogram using a ResNet-18 model, bypassing per-frame detection and tracking. An echogram is a two-dimensional image in which time runs horizontally and each column encodes the strongest sonar echo at each range for one video frame, so a few hundred frames collapse into one image. With weak labels from an existing detector-tracker and strong human labels, plus domain-specific augmentations, the model reaches a normalized mean absolute error of 23 percent on the KL-val split and 30.7 percent on the right-bank test set. Error is much larger for downstream-moving fish, which are rare in the training data, and the model undercounts in clips with many overlapping fish. The authors position this as a feasibility result: the accuracy is higher than state-of-the-art tracking pipelines but comparable to early proof-of-concept counts, while the compute per window is much lower.

Load-bearing premise

The assumption that generating an echogram is cheap enough to preserve the method's compute advantage: the paper's advertised single pass per 200 frames does not count the background subtraction and connected-components preprocessing per video frame, and that cost is not measured.

Editorial extensions

If this is right

Counting cost drops to one lightweight forward pass per 200-frame window, enabling deployment on low-compute or offline hardware at remote river sites.
Weak labels alone almost match strong labels on out-of-distribution data, so previously unlabeled sonar footage can be harnessed for training.
Domain-specific augmentation, especially the realistic horizontal flip, improves accuracy while naive flipping hurts, giving a recipe for similar temporal-compression counting tasks.
The largest remaining error is downstream fish due to class imbalance, so reducing that imbalance is the main lever for improving total counts.
At 23 percent error the method is not yet a replacement for tracking-based counts, but it is a viable low-cost screening tool or complement to manual review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the advertised compute advantage depends on echogram generation being cheap; the reported timings omit background subtraction and connected-components preprocessing, so a fair end-to-end comparison on field hardware could change the conclusion.
The temporal-compression idea may transfer to other sonar counting tasks, such as eel or trout passage, if the weak-label and augmentation recipe carries over.
The systematic undercounting at high fish density suggests that a detection-free regressor could benefit from density-aware losses or multiple time-resolution windows, a testable extension not explored here.
Since the model uses only intensity and lateral-position channels, adding beam-level or motion-direction information might specifically improve downstream counting, which the current setup handles worst.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look: this is the first paper I've seen that regresses salmon counts directly from echograms, skipping detection and tracking entirely, and the basic feasibility claim holds up. The numbers are real but need some scrutiny before you trust them.

What's genuinely new: the echogram representation (max-intensity projection over beams plus a lateral-position channel) plus a ResNet-18 regressing counts for 200-frame windows. That's a clean, lightweight pipeline: one forward pass per window, no per-frame detector. The realistic horizontal flip augmentation—flipping and then inverting the lateral-position channel—is a small, correct domain-specific contribution. And the weak-supervision protocol, labels from the CFC detector-tracker, is not circular because the final evaluation is against human strong counts; it's a sensible way to exploit 33k unlabeled clips.

Credit where due: the paper is honest about its limitations and compares against the strong detector-tracker baselines. The observation that downstream fish are badly underrepresented (175 vs 8 in KL-val) is stated plainly.

Soft spots, in order of severity. First, the headline 23% nMAE is on a 66-clip validation set that the authors used for early stopping and threshold selection. That is an optimistic point estimate, with no error bars or repeated splits. It could easily be 30% on a true held-out set; the out-of-distribution 30.7% is actually more informative. Second, downstream counts are bad: 37.5% in-distribution and 96% out-of-distribution. That is not a minor blemish; escapement includes downstream movers. Third, the compute advantage is partially unearned: echogram generation still runs per-frame background subtraction and connected components, and the paper doesn't measure that cost. If that preprocessing is slow on field laptops, the lightweight claim weakens. Fourth, no code or data are released, so the numbers can't be checked independently.

These don't sink the central idea. The paper is a proof of concept, not a deployment-ready system, and it reads like one. The experiments support feasibility, the ablation table is coherent, and the honest discussion of underprediction at high density is a good sign.

Recommendation: send it to review. A serious referee should ask for error bars or a held-out test set, a measurement of echogram generation time, and disaggregated results by density. But the work deserves referee time; it's new, the writing is clear, and the limitations are stated rather than buried.

Referee Report

4 major / 5 minor

Summary. The paper proposes a method to count migrating salmon directly from echograms—two-dimensional temporal max-intensity projections of multi-beam sonar video—using a ResNet-18 regressor that outputs upstream and downstream counts per 200-frame window. Training combines human-validated 'strong' labels from the Caltech Fish Counting dataset with weakly supervised labels generated by the authors' own detector-tracker pipeline. The authors introduce domain-specific augmentations and report an nMAE of 23% on KL-val and 30.7% on KR, claiming this as a low-compute alternative to per-frame detection and tracking.

Significance. If the numbers hold, the work is a useful feasibility demonstration: counting regression on a compressed temporal representation is markedly cheaper at inference than per-frame YOLO plus tracking, and echograms are already the native review medium for sonar technicians. Positive contributions include the use of the public CFC benchmark, a direct comparison against prior tracking-by-detection results, clear ablations, and a domain-motivated 'realistic horizontal flip' augmentation. The central caveat is that the primary result is measured on the development set and lacks uncertainty quantification; the compute motivation is also not yet demonstrated end-to-end. As a feasibility study, the idea is worth pursuing, but the current evidence is conditional.

major comments (4)

[Sec. 3.2, Sec. 5.2, Table 2] The headline 23% nMAE on KL-val is not an independent test statistic. KL-val is used for early stopping (Sec. 3.2) and for selecting echogram generation thresholds (Sec. 5.2, Table 2), and the final model is the result of choices made by looking at this set. With only N=66 clips, the reported number is likely to be optimistically biased. Please report results on a properly held-out split, for example repeated cross-validation or a separate validation set, or present KR as the primary held-out result with confidence intervals.
[Sec. 5, Tables 1–3] All reported nMAE values are single point estimates without error bars, repeated seeds, or bootstrap intervals. The ablations in Table 3 differ by only a few points (e.g., 43.2 vs. 39.3 on KL-train with different augmentations), and without variance estimates it is impossible to tell which differences are meaningful. Please provide confidence intervals or results across multiple random seeds.
[Sec. 3.1 and Introduction] The claimed compute advantage is not quantified. The method still requires per-frame background subtraction at three thresholds and connected-component analysis for every frame before the echogram is constructed; the paper does not measure the runtime or energy cost of this preprocessing, nor does it compare end-to-end cost against the detector-tracker. The statement 'requiring only a single forward pass every 200 frames' omits the cost of building the input. Please measure end-to-end processing time on representative field hardware.
[Sec. 4.1, Sec. 5, Tables 1 and 3] The downstream count errors are very large: 37.5% nMAE on KL-val and 96% on KR for the best model, while the abstract reports a single 23% aggregate. Since downstream fish are part of the management goal, the aggregate metric obscures a failure mode. Additionally, the weak labels are produced by the same CFC detector-tracker used as the baseline, so the model may inherit the tracker's biases (especially for downstream fish); the paper does not analyze weak-label noise. Please report downstream and upstream errors separately in the abstract-level claims and provide a characterization of weak-label bias.

minor comments (5)

[Sec. 4.2, Eq. (1)] The summation in Eq. (1) is written from i=0 to N, but with N clips the index should start at 1.
[Sec. 4.1] The sentence 'We ensure there is no overlap between the KL validation set and the detector-tracker annotated training or validation set' is unclear; please specify how the 33,437 weak-labeled clips were split between training and validation.
[Table 3] The use of bullet points to indicate augmentation settings makes the table hard to parse; a table with explicit True/False columns would be clearer.
[Figure 2] The caption does not explain how ground-truth counts are binned to determine dot sizes; please add a short description.
[References] Reference [11] lists 'Trout Unlimited Caltech' as an author; this appears to be an organization rather than a person and should be formatted consistently with the other references.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim is evaluated against independent human-labeled counts, and the self-citations are not load-bearing.

full rationale

The paper's derivation chain is: generate echograms from sonar video, train a ResNet-18 regressor to predict left/right counts from 200-frame echogram slices, and evaluate nMAE against the CFC strong labels in KL-val and KR. The central evaluation is external to the fitted model: the KL-val and KR counts come from the CFC benchmark's human-verified strong labels, not from the model's own outputs or from equations that presuppose the target counts. The weak labels in Section 4.1 are indeed generated by the authors' own CFC detector-tracker pipeline, which is a self-referential training-data choice, but the model is not evaluated on those weak labels and the reported 23% error is measured against the independent strong labels. The self-citations to [8] and [11] are dataset/baseline references and are not used to force the method's conclusion; no uniqueness theorem, ansatz-by-citation, or renaming of a known result appears. The echogram-generation thresholds are tuned on validation performance, which is standard hyperparameter selection rather than a fitted-input-called-prediction step. Overall, no specific step reduces by construction to its own input, so the paper exhibits no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its contributions rest on preprocessing thresholds chosen by trial and error, a standard transfer-learning backbone, and weak labels generated by the authors' own earlier detector-tracker pipeline.

free parameters (2)

Echogram background subtraction thresholds (alpha0, alpha1, alpha2) and size threshold = alpha0=20, alpha1=40, alpha2=100, size threshold=100 (best row in Table 2)
Tuned by trial and error. Table 2 shows nMAE swings from 84.7 to 23.0 across threshold settings, so the central result depends on this preprocessing choice.
Model hyperparameters (learning rate, batch size, input size, max epochs) = 1e-5, 256, 200x800, 100
Chosen by hand and reported for reproducibility. Early stopping is based on the same KL-val set whose nMAE is the headline result.

assumptions (4)

domain assumption ImageNet-pretrained ResNet-18 transfers to normalized sonar echograms without architectural change.
Section 3.2 fine-tunes an ImageNet-pretrained ResNet-18; the assumption is that natural-image features are useful for echogram texture patterns.
domain assumption Background subtraction and connected-components thresholds isolate fish from noise without removing relevant tracks.
Section 3.1 tunes thresholds until a 'qualitatively acceptable' echogram is produced; Table 2 shows performance is highly sensitive to these thresholds.
domain assumption The beam-max projection plus lateral-position channel retains enough information to count fish.
Section 3.1 collapses each multi-beam frame into a single column with max intensity and normalized lateral position; overlapping fish in range may be lost in this compression.
domain assumption Weak labels from the CFC detector-tracker are sufficiently accurate to serve as training supervision.
Section 4.1 uses 33,437 images labeled by the public pipeline; any systematic detector bias, such as missed overlapping fish, would be inherited by the trained regressor.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Counting Fish with Temporal Representations of Sonar Video." pith.science (2026). https://pith.science/paper/VV25RKWJ

@misc{pith2026250205129,
  author       = {Pith},
  title        = {Pith review of: Counting Fish with Temporal Representations of Sonar Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VV25RKWJ}},
  note         = {Machine review of arXiv:2502.05129}
}

read the original abstract

Accurate estimates of salmon escapement - the number of fish migrating upstream to spawn - are key data for conservation and fishery management. Existing methods for salmon counting using high-resolution imaging sonar hardware are non-invasive and compatible with computer vision processing. Prior work in this area has utilized object detection and tracking based methods for automated salmon counting. However, these techniques remain inaccessible to many sonar deployment sites due to limited compute and connectivity in the field. We propose an alternative lightweight computer vision method for fish counting based on analyzing echograms - temporal representations that compress several hundred frames of imaging sonar video into a single image. We predict upstream and downstream counts within 200-frame time windows directly from echograms using a ResNet-18 model, and propose a set of domain-specific image augmentations and a weakly-supervised training protocol to further improve results. We achieve a count error of 23% on representative data from the Kenai River in Alaska, demonstrating the feasibility of our approach.

Figures

Figures reproduced from arXiv: 2502.05129 by the authors.

**Figure 1.** Clockwise: 1) a frame of the raw ARIS file; 2) the same frame after applying background subtraction with a minimum positive [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Mean and standard deviation of total predicted counts vs total ground truth counts per clip on the KL-val and KR test sets. Size of the dot corresponds to the number of images with the associated ground truth count. The model systematically predicts lower counts than ground truth for KR clips with large numbers of fish, where tracks of distinct fish may overlap and become difficult to distinguish on the echogram. po… view at source ↗

**Figure 3.** Left: depiction of the horizontal plane of a multi-beam [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: ARIS display software used by sonar technicians [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

[1]

Non-sonar estimating tools

Alaska Department of Fish and Game. Non-sonar estimating tools. https://www.adfg.alaska.gov/index. cfm ? adfg = sonar . nonsonartools, n.d.. Alaska Fisheries Sonar. 1

work page
[2]

Sonar tools: Imaging sonar

Alaska Department of Fish and Game. Sonar tools: Imaging sonar. https://www.adfg.alaska.gov/index. cfm ? adfg = sonar . didson, n.d.. Alaska Fisheries Sonar. 1, 2

work page
[3]

Simple online and realtime tracking

Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP) , pages 3464–3468. IEEE, 2016. 2

work page 2016
[4]

Validation of fish length esti- mations from a high frequency multi-beam sonar (aris) and its utilisation as a field-based measurement technique

Denham Cook, Karen Middlemiss, Peter Jaksons, William Davison, and Alistair Jerrett. Validation of fish length esti- mations from a high frequency multi-beam sonar (aris) and its utilisation as a field-based measurement technique. Fish- eries Research, 218:59–68, 2019. 1

work page 2019
[5]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1

2016
[6]

dual-frequency identification sonar

John A Holmes, George MW Cronkite, Hermann J En- zenhofer, and Timothy J Mulligan. Accuracy and preci- sion of fish-count data from a “dual-frequency identification sonar”(didson) imaging system. ICES Journal of Marine Sci- ence, 63(3):543–555, 2006. 1

work page 2006
[7]

Unsuper- vised domain adaptation in the real world: A case study in sonar video

Justin Kay, Suzanne Stathatos, Siqi Deng, Erik Young, Pietro Perona, Sara Beery, and Grant Van Horn. Unsuper- vised domain adaptation in the real world: A case study in sonar video. In NeurIPS 2023 Computational Sustainability: Promises and Pitfalls from Theory to Deployment. 2

work page 2023
[8]

The caltech fish counting dataset: A benchmark for multiple- object tracking and counting, 2022

Justin Kay, Peter Kulits, Suzanne Stathatos, Siqi Deng, Erik Young, Sara Beery, Grant Van Horn, and Pietro Perona. The caltech fish counting dataset: A benchmark for multiple- object tracking and counting, 2022. 1, 2, 3

work page 2022

Show all 20 references

[9]

Align and distill: Unifying and improving domain adaptive object detection

Justin Kay, Timm Haucke, Suzanne Stathatos, Siqi Deng, Erik Young, Pietro Perona, Sara Beery, and Grant Van Horn. Align and distill: Unifying and improving domain adaptive object detection. arXiv preprint arXiv:2403.12029, 2024. 2

2024 arXiv
[10]

Key, James D

Brandon H. Key, James D. Miller, Steven J. Fleischman, and Jiaqi Huang. Chinook salmon passage in the kenai river at river mile 13.7 using adaptive resolution imaging sonar,
[11]

Au- tomated salmonid counting in sonar data

Peter Kulits, Angelina Pan, SM Beery, Erik Young, Pietro Perona, Grant Van Horn, and Trout Unlimited Caltech. Au- tomated salmonid counting in sonar data. In NeurIPS 2020 Workshop on tackling climate change with machine learning,

2020
[12]

Yolo-based fish detection in underwater environments

Mohammed Yasser Ouis and Moulay Akhloufi. Yolo-based fish detection in underwater environments. Environmental Sciences Proceedings, 29(1):44, 2023. 2

2023
[13]

You only look once: Unified, real-time object de- tection

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object de- tection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016. 2

2016
[14]

Aris sonars

Sound Metrics. Aris sonars. http : / / www . soundmetrics . com / Products / ARIS - Sonars/,
[17]

In narrow, shallow streams, constructing weirs (fences with a gate controlled by a tech- nician) allows technicians to count salmon one by one as they pass through

Salmon monitoring additional information Several methods exist for monitoring salmonid escapement, each with their own trade-offs. In narrow, shallow streams, constructing weirs (fences with a gate controlled by a tech- nician) allows technicians to count salmon one by one as ...
[18]

The available transformations are described be- low in order of application

Data preprocessing All images in the training, validation, and test sets are sub- ject to a sequence of transformations to standardize the in- put format. The available transformations are described be- low in order of application
[19]

Shifts all pixel values (initially lying between 0 and 1) down by 0.5 and divides the result by 0.25

Shift and rescale all channels. Shifts all pixel values (initially lying between 0 and 1) down by 0.5 and divides the result by 0.25
[20]

Figure 4

Resize to 200 by 800 pixels. Figure 4. ARIS display software used by sonar technicians showing the echogram view and corresponding frame in sonar video [10]
[2015]

Technical Report 17-33, Alaska Department of Fish and Game, Divisions of Sport Fish and Commercial Fish- eries, 2017. 1

2017
[2024]

2 Counting Fish with Temporal Representations of Sonar Video Supplementary Material Figure 3

Accessed: 2024-08-14. 2 Counting Fish with Temporal Representations of Sonar Video Supplementary Material Figure 3. Left: depiction of the horizontal plane of a multi-beam sonar configuration; right: camera placement on the left and right banks of the Kenai river in Alaska [10]

2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Non-sonar estimating tools

Alaska Department of Fish and Game. Non-sonar estimating tools. https://www.adfg.alaska.gov/index. cfm ? adfg = sonar . nonsonartools, n.d.. Alaska Fisheries Sonar. 1

work page

[2] [2]

Sonar tools: Imaging sonar

Alaska Department of Fish and Game. Sonar tools: Imaging sonar. https://www.adfg.alaska.gov/index. cfm ? adfg = sonar . didson, n.d.. Alaska Fisheries Sonar. 1, 2

work page

[3] [3]

Simple online and realtime tracking

Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP) , pages 3464–3468. IEEE, 2016. 2

work page 2016

[4] [4]

Validation of fish length esti- mations from a high frequency multi-beam sonar (aris) and its utilisation as a field-based measurement technique

Denham Cook, Karen Middlemiss, Peter Jaksons, William Davison, and Alistair Jerrett. Validation of fish length esti- mations from a high frequency multi-beam sonar (aris) and its utilisation as a field-based measurement technique. Fish- eries Research, 218:59–68, 2019. 1

work page 2019

[5] [5]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1

2016

[6] [6]

dual-frequency identification sonar

John A Holmes, George MW Cronkite, Hermann J En- zenhofer, and Timothy J Mulligan. Accuracy and preci- sion of fish-count data from a “dual-frequency identification sonar”(didson) imaging system. ICES Journal of Marine Sci- ence, 63(3):543–555, 2006. 1

work page 2006

[7] [7]

Unsuper- vised domain adaptation in the real world: A case study in sonar video

Justin Kay, Suzanne Stathatos, Siqi Deng, Erik Young, Pietro Perona, Sara Beery, and Grant Van Horn. Unsuper- vised domain adaptation in the real world: A case study in sonar video. In NeurIPS 2023 Computational Sustainability: Promises and Pitfalls from Theory to Deployment. 2

work page 2023

[8] [8]

The caltech fish counting dataset: A benchmark for multiple- object tracking and counting, 2022

Justin Kay, Peter Kulits, Suzanne Stathatos, Siqi Deng, Erik Young, Sara Beery, Grant Van Horn, and Pietro Perona. The caltech fish counting dataset: A benchmark for multiple- object tracking and counting, 2022. 1, 2, 3

work page 2022