REVIEW 4 major objections 4 minor 1 cited by
Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Grounded vision-language retrieval identifies fish species without retraining, using an image-keyed vector store to beat a trained baseline.
desk verdict A plausible training-free RAG boost for fish classification, but the evaluation is too thin and the split-leakage risk needs checking before the numbers are trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the image-based vector store: reference images of fish species are embedded with the CLIP visual encoder and stored as keys, and the test image is embedded with the same encoder so that nearest-neighbour search retrieves species descriptions that are then concatenated as tokens into a pretrained LLaVA 1.5 language model. Because the query encoder and the stored keys share one embedding space, retrieval and generation are aligned without any training, and the retrieved text acts as grounded external knowledge that constrains the otherwise open-ended visual question.
What would settle it
Compute nearest-neighbour retrieval accuracy on the FishNet test set after removing one species from the vector store; if a species absent from the store is not retrieved correctly, the paper's claim that grounding enables identification of unseen species fails. Even within the closed-store setup, measuring pairwise CLIP embedding distances and showing that same-species nearest-neighbour accuracy is near chance would overturn the load-bearing assumption.
Extended reading notes
Core claim
The paper's central claim is that visual RAG with an image-embedding-keyed vector store yields 0.8403 final-prediction accuracy and 0.8684 retrieval accuracy on FishNet fish categorization, outperforming an ImageNet-pretrained InceptionV3 baseline (0.7501) while using no domain-specific training or knowledge of the task. The authors argue this is the first RAG knowledge base keyed by image embeddings rather than text, motivated by the fact that direct text-to-image similarity over low-resolution, partly occluded vessel imagery is noisy. They demonstrate the necessity of grounding with an ablation in which the same LLaVA model, given only the category list and no retrieved description, misclassifies the example fish, whereas the RAG-augmented prompt identifies the tuna from its torpedo-shaped body and metallic blue colour.
Load-bearing premise
The method assumes CLIP embeddings put images of the same fish species closer together than images of different species, so that nearest-neighbour retrieval over reference images returns the correct species description for a test image; the paper's own PCA visualization shows categories heavily mixed in embedding space, and the setup never tests a species that is absent from the vector store.
Editorial extensions
If this is right
- Fish classification from on-deck video can be performed without retraining for a new vessel environment, since the retrieval store can be extended or swapped at inference time.
- Because retrieval already reaches 0.8684 top-1 accuracy and 0.9781 top-3 accuracy, improving the generation step through instruction tuning or prompt optimization should push final prediction accuracy above retrieval accuracy.
- The same modular image-embedding-keyed RAG structure could apply to other non-text marine data streams, such as acoustic or satellite imagery, wherever a small labeled reference set exists.
- Grounding at inference time gives a concrete mechanism for handling rare or long-tailed classes: the model looks up a description for the rare class instead of having to memorize it during training.
Reading between the lines
- If the image-embedding-keyed store is what carries the gain, then the same modular RAG structure should transfer to other non-text modalities, such as audio or satellite imagery, where a small labeled reference set exists but supervised fine-tuning is impractical.
- The paper reports category-level accuracy with species nested inside categories; evaluating species-level retrieval accuracy would likely lower the numbers and reveal where the CLIP embedding space actually fails, since the authors note that species within a category are hard to distinguish.
- A direct test of the 'unseen species' motivation would be to drop one species from the vector store at a time and measure retrieval accuracy on it; the current evaluation cannot support the claim that unseen species are identified, because every test category is present in the store.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a retrieval-augmented generation (RAG) pipeline for open-domain fish species classification in on-deck fishing video. A CLIP visual encoder embeds a reference set of FishNet validation images into a vector store keyed by image embeddings; at test time, similarity search retrieves species descriptions that are fed to a pre-trained LLaVA model as context for answering the classification question. On five FishNet categories the method reports 0.8684 retrieval accuracy and 0.8403 final-prediction accuracy, outperforming an ImageNet-pretrained InceptionV3 baseline (0.7501), without any fine-tuning of the VLM. The paper frames this as a preliminary demonstration of bottom-up, open-domain vision for marine monitoring, with generalizability, adaptivity, and the ability to handle unseen species as motivating claims.
Significance. If the central empirical result holds, the paper makes a useful preliminary contribution: it shows that a frozen vision-language model combined with an image-keyed retrieval store can, without task-specific training, provide competitive classification on a niche, low-quality-imagery domain. The method is simple, modular, and potentially scalable, and the idea of using image embeddings rather than text keys for the vector store is worth reporting. The authors are explicit that this is preliminary work and that many components (re-ranking, prompt optimization, fine-tuning, multi-query search) are left to future work. However, the current experimental support is thin in several load-bearing respects, including possible frame-level leakage between the vector store and test set, lack of statistical uncertainty estimates, and claims about unseen species that the retrieval mechanism cannot actually support as stated.
major comments (4)
- [§3.1 and Appendix A.3] The vector store is built from FishNet validation-set embeddings and accuracy is measured on test-set samples, but the paper does not state whether the official split is frame-level, video-level, or trip-level, nor how many unique videos are present. If the split is frame-level, near-duplicate frames from the same recording can appear in both the store and the test set; given the high temporal redundancy of on-deck video, this would inflate both the reported retrieval accuracy (0.8684) and final-prediction accuracy (0.8403) in Table 1. The authors should report the exact split semantics, the number of unique videos, and a nearest-neighbor distance analysis between store and test embeddings to rule out trivial same-video retrieval.
- [§1 and §3] The paper repeatedly claims that the method enables accurate identification of 'unseen species,' but the proposed pipeline retrieves descriptions from a vector store containing reference images of the target species; a species absent from the store cannot produce the correct retrieved description. No experiment holds out entire species from the store. The claim should be restricted to novel instances of species already present in the reference set, or the method must be evaluated with species completely absent from the store.
- [Table 1 and §4] All accuracies are reported as single runs with no error bars, confidence intervals, or significance tests, for a difference of 0.0902 between the proposed method (0.8403) and the baseline (0.7501). The authors should provide averages over multiple runs or bootstrap resampling, specify the number of test samples, and, if possible, a statistical test. In addition, no hyperparameters are given for the retrieval step (top-k), the CLIP variant, the LLaVA version, or the prompting scheme, which prevents independent reproduction of the central result.
- [§4 and Table 1] The baseline comparison is limited to a single ImageNet-pretrained InceptionV3, and the no-RAG ablation mentioned in the text is not quantified as a top-1 accuracy in Table 1. To support the claim that retrieval grounding is responsible for the observed gain, the paper should report quantitative accuracy for a zero-shot LLaVA baseline without RAG and for a stronger fine-tuned vision model, rather than only a qualitative example in Fig. 2.
minor comments (4)
- [Figure 2] The caption states that the retrieved description is 'not shown in figure,' but the example output says the fish is a tuna; the without-retrieval example says Mahi Mahi. Clarify what category list was provided in each case so the reader can compare the two settings.
- [Appendix A.3] Figure 5 is used to argue that embeddings are mixed and the task is difficult, but a 2D PCA plot is not quantitative evidence. Consider reporting per-class nearest-neighbor statistics or class separability measures.
- [Abstract and §5] The statement that the system works 'without domain-specific training or knowledge of the task itself' is too strong: the vector store is built from labeled validation images and the retrieved text descriptions encode task-specific species knowledge. Soften this claim.
- [Throughout] The model name 'LLaV A' appears with a spurious space in several places; standardize to LLaVA.
Circularity Check
No significant circularity: the pipeline is an empirical retrieval evaluation with no fitted parameters and no derivation that reduces to its inputs.
full rationale
The paper's method is a retrieval-augmented classification pipeline: CLIP embeddings of FishNet validation images are stored as keys in a vector database, nearest-neighbor search retrieves the corresponding species description, and a pretrained LLaVA model uses that description to answer the classification prompt. There is no fitted parameter, no equation defining the final prediction in terms of the retrieval score, and no self-citation chain. The vector store is explicitly built from the validation set, while accuracy is reported on test-set samples (Sec. 3.1 and App. A.3), so the reported numbers are empirical measurements rather than identities. The claim about identifying unseen species is an overclaim rather than a circular computation, since a species absent from the store cannot be retrieved; that is a correctness/scope concern, not circularity. The possible near-duplicate-frame overlap between FishNet's official validation and test splits would be benchmark contamination, not a logical reduction of the result to its inputs. The baseline is an external ImageNet-pretrained InceptionV3, and the paper's own Fig. 5 distinguishes vector-store samples from test-set samples. Consequently, no circular step can be exhibited and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- retrieval top-k =
not reported
- species description text =
not provided
assumptions (3)
- domain assumption Pretrained CLIP embeddings are discriminative enough for nearest-neighbor fish-species retrieval on low-quality on-deck images.
- domain assumption LLaVA can correctly integrate a retrieved textual species description with the query image to produce a correct classification.
- domain assumption The FishNet validation set is disjoint from the test set and its labels are reliable.
Cite this review
Pith. "Pith review of Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation." pith.science (2026). https://pith.science/paper/K76I4PIT
@misc{pith2026241202262,
author = {Pith},
title = {Pith review of: Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation},
year = {2026},
howpublished = {\url{https://pith.science/paper/K76I4PIT}},
note = {Machine review of arXiv:2412.02262}
}
read the original abstract
Climate change's destruction of marine biodiversity is threatening communities and economies around the world which rely on healthy oceans for their livelihoods. The challenge of applying computer vision to niche, real-world domains such as ocean conservation lies in the dynamic and diverse environments where traditional top-down learning struggle with long-tailed distributions, generalization, and domain transfer. Scalable species identification for ocean monitoring is particularly difficult due to the need to adapt models to new environments and identify rare or unseen species. To overcome these limitations, we propose leveraging bottom-up, open-domain learning frameworks as a resilient, scalable solution for image and video analysis in marine applications. Our preliminary demonstration uses pretrained vision-language models (VLMs) combined with retrieval-augmented generation (RAG) as grounding, leaving the door open for numerous architectural, training and engineering optimizations. We validate this approach through a preliminary application in classifying fish from video onboard fishing vessels, demonstrating impressive emergent retrieval and prediction capabilities without domain-specific training or knowledge of the task itself.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Environmental large language model Evaluation (ELLE) dataset: A Benchmark for Evaluating Generative AI applications in Eco-environment Domain
A new 1,130-question benchmark, ELLE-QA, is proposed as the first standard test of AI language models in the environmental and ecological sciences.
Reference graph
Works this paper leans on
-
[1]
Wild salmon enumeration and monitoring using deep learning empow- ered detection and tracking
William I. Atlas et al. “Wild salmon enumeration and monitoring using deep learning empow- ered detection and tracking”. In: Frontiers in Marine Science 10 (2023). ISSN : 2296-7745. URL: https://www.frontiersin.org/articles/10.3389/fmars.2023.1200408
arXiv 2023
-
[2]
Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Mul- timodal LLMs
Davide Caffagni et al. Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Mul- timodal LLMs. May 22, 2024. arXiv: 2404.15406[cs]. URL: http://arxiv.org/abs/ 2404.15406
arXiv 2024
-
[3]
An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale
Alexey Dosovitskiy et al. “An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale”. In: International Conference on Learning Representations. Oct. 2, 2020. URL: https://openreview.net/forum?id=YicbFdNTTy
work page 2020
-
[4]
Knowledge Augmented Instruction Tuning for Zero-shot Animal Species Recognition
Zalan Fabian et al. “Knowledge Augmented Instruction Tuning for Zero-shot Animal Species Recognition”. In: NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. Nov. 26, 2023. URL: https://openreview.net/forum?id=OQHckRYbpT
work page 2023
-
[5]
Joint SDG Fund | Goal 14: Life below water
Goal 14 : Life below water . Joint SDG Fund | Goal 14: Life below water. URL: https : //jointsdgfund.org/sustainable- development- goals/goal- 14- life- below- water
-
[6]
REALM: Retrieval-Augmented Language Model Pre-Training
Kelvin Guu et al. REALM: Retrieval-Augmented Language Model Pre-Training. Feb. 10, 2020. arXiv: 2002.08909[cs]. URL: http://arxiv.org/abs/2002.08909
arXiv 2020
-
[7]
Deep Residual Learning for Image Recognition
Kaiming He et al. Deep Residual Learning for Image Recognition . Dec. 10, 2015. arXiv: 1512.03385[cs]. URL: http://arxiv.org/abs/1512.03385
arXiv 2015
- [8]
Show all 31 references
-
[9]
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi- Source Multimodal Knowledge Memory
Ziniu Hu et al. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi- Source Multimodal Knowledge Memory. Apr. 3, 2023. arXiv: 2212.05221[cs]. URL: http: //arxiv.org/abs/2212.05221
2023 arXiv
-
[10]
Active Retrieval Augmented Generation
Zhengbao Jiang et al. Active Retrieval Augmented Generation. Oct. 21, 2023. arXiv: 2305. 06983[cs]. URL: http://arxiv.org/abs/2305.06983
2023 arXiv
-
[11]
FathomNet: A global image database for enabling artificial intelligence in the ocean
Kakani Katija et al. “FathomNet: A global image database for enabling artificial intelligence in the ocean”. In:Scientific Reports 12.1 (Sept. 23, 2022). Number: 1 Publisher: Nature Publishing Group, p. 15914. ISSN : 2045-2322. URL: https://www.nature.com/articles/s41598- 022-19939-2
2022
-
[12]
The Fishnet Open Images Database: A Dataset for Fish Detec- tion and Fine-Grained Categorization in Fisheries
Justin Kay and Matt Merrifield. The Fishnet Open Images Database: A Dataset for Fish Detec- tion and Fine-Grained Categorization in Fisheries. June 16, 2021. arXiv: 2106.09178[cs]. URL: http://arxiv.org/abs/2106.09178
2021 arXiv
-
[13]
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. “ImageNet Classification with Deep Convolutional Neural Networks”. In: Advances in Neural Information Processing Systems. V ol. 25. Curran Associates, Inc., 2012.URL: https://papers.nips.cc/paper_files/ paper/2012/hash/c3...
2012
-
[14]
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Junnan Li et al. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. June 15, 2023. arXiv: 2301.12597[cs]. URL: http: //arxiv.org/abs/2301.12597
2023 arXiv
-
[15]
Grounded Language-Image Pre-training
Liunian Harold Li et al. “Grounded Language-Image Pre-training”. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, June 2022, pp. 10955–10965. ...
2022
-
[16]
Visual Instruction Tuning
Haotian Liu et al. Visual Instruction Tuning. Dec. 11, 2023. arXiv: 2304.08485[cs]. URL: http://arxiv.org/abs/2304.08485. 5
2023 arXiv
-
[17]
KRISP: Integrating Implicit and Symbolic Knowledge for Open- Domain Knowledge-Based VQA
Kenneth Marino et al. “KRISP: Integrating Implicit and Symbolic Knowledge for Open- Domain Knowledge-Based VQA”. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashvill...
2021
-
[18]
OK-VQA: A Visual Question Answering Benchmark Requiring Exter- nal Knowledge
Kenneth Marino et al. “OK-VQA: A Visual Question Answering Benchmark Requiring Exter- nal Knowledge”. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019). Conference Name: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognitio...
2019
-
[19]
New frontiers in AI for biodiversity research and conservation with multimodal language models
Zhongqi Miao et al. “New frontiers in AI for biodiversity research and conservation with multimodal language models”. In: (Aug. 1, 2024). Publisher: EcoEvoRxiv. URL: https : //ecoevorxiv.org/repository/view/7477/#!
2024
-
[20]
A deep active learning system for species identification and counting in camera trap images
Mohammad Sadegh Norouzzadeh et al. A deep active learning system for species identification and counting in camera trap images. Oct. 21, 2019. arXiv: 1910.09716[cs,eess,stat] . URL: http://arxiv.org/abs/1910.09716
2019 arXiv
-
[21]
Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning
Mohammad Sadegh Norouzzadeh et al. “Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning”. In: Proceedings of the National Academy of Sciences 115.25 (June 19, 2018). Publisher: Proceedings of the National Academy of Science...
2018 doi
-
[22]
Learning Transferable Visual Models From Natural Language Supervi- sion
Alec Radford et al. “Learning Transferable Visual Models From Natural Language Supervi- sion”. In: Proceedings of the 38th International Conference on Machine Learning. International Conference on Machine Learning. ISSN: 2640-3498. PMLR, July 1, 2021, pp. 8748–8763. URL: https...
2021
-
[23]
Rita Ramos et al.SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmenta- tion. Mar. 28, 2023. arXiv: 2209.15323[cs]. URL: http://arxiv.org/abs/2209.15323
2023 arXiv
-
[24]
You Only Look Once: Unified, Real-Time Object Detection
Joseph Redmon et al. “You Only Look Once: Unified, Real-Time Object Detection”. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, June 2016, pp. 779–788...
2016
-
[25]
Retrieval-Augmented Transformer for Image Captioning
Sara Sarto et al. Retrieval-Augmented Transformer for Image Captioning . Aug. 22, 2022. arXiv: 2207.13162[cs]. URL: http://arxiv.org/abs/2207.13162
2022 arXiv
-
[26]
K-LITE: Learning Transferable Visual Models with External Knowl- edge
Sheng Shen et al. “K-LITE: Learning Transferable Visual Models with External Knowl- edge”. In: Advances in Neural Information Processing Systems 35 (Dec. 6, 2022), pp. 15558– 15573. URL: https : / / papers . nips . cc / paper _ files / paper / 2022 / hash / 63fef0802863f47775c...
2022
-
[27]
BioCLIP: A Vision Foundation Model for the Tree of Life
Samuel Stevens et al. “BioCLIP: A Vision Foundation Model for the Tree of Life”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, pp. 19412–19424
2024
-
[28]
The iNaturalist Species Classification and Detection Dataset
Grant Van Horn et al. “The iNaturalist Species Classification and Detection Dataset”. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT: IEEE, June 2018, pp. 8...
2018
-
[29]
Advancing artificial intelligence in fisheries requires novel cross-sector collaborations
Kate Wing and Benjamin Woodward. “Advancing artificial intelligence in fisheries requires novel cross-sector collaborations”. In: ICES Journal of Marine Science(Aug. 28, 2024). Ed. by Howard Browman, fsae118. ISSN : 1054-3139, 1095-9289. URL: https://academic.oup. com/icesjms/...
2024 doi
-
[30]
Multi-Modal Answer Validation for Knowledge-Based VQA
Jialin Wu et al. Multi-Modal Answer Validation for Knowledge-Based VQA. Dec. 13, 2021. arXiv: 2103.12248[cs]. URL: http://arxiv.org/abs/2103.12248
2021 arXiv
-
[31]
MSR-VTT: A Large Video Description Dataset for Bridging Video and Lan- guage
Jun Xu et al. “MSR-VTT: A Large Video Description Dataset for Bridging Video and Lan- guage”. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, June 2016...
2016
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.