{"id":"f19010b9-d39e-4c67-9133-52a4157b38ea","arxiv_id":"2509.05703","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Progressive knowledge augmentation with VLM-extracted pattern descriptions improves marine mammal spectrogram classification over vanilla VLMs (25.4% vs 13.2%) while keeping models retraining-free.","lead":"This paper tests whether vision language models, given a growing library of written pattern descriptions, can identify marine mammal species from sound spectrograms without retraining. The method reaches 25.4% accuracy on up to 24 species, a 92% relative gain over a plain VLM, and produces interpretable descriptions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"92% improvement is confounded by supervised retrieval: no control separates labeled-example access from the VLM-generated text knowledge base.","rationale":"The reader's weakest_assumption identifies the semantic gap between spectrogram images and text descriptions as the risk. I agree that is a real risk, but I think the more load-bearing issue is attribution: the experimental design cannot distinguish the proposed 'progressive knowledge accumulation' from a generic supervised-retrieval effect. Algorithm 1 stores labeled training examples in the KB. The vanilla baseline has no labels; the fixed baseline has unlabeled generic text. Therefore the 92% number is not a clean test of the hypothesis. The UMAP evidence cited by the reader is actually compatible with both readings: if text descriptions cluster by linguistic similarity rather than species, that makes the text bottleneck more likely, but the missing raw-embedding control means we don't know whether the text patterns are even necessary. The r=-0.44 negative correlation further complicates the 'more knowledge helps' story. Because the decisive control is simple and well-defined, I would not reject the paper, but I would not accept the central attribution as established; the verdict should be conditional on (or unverified until) the control is run. I mark partial agreement with the reader because the semantic-gap concern is adjacent but not identical to the supervised-retrieval confound.","tokens_in":5015,"tokens_out":11750,"duration_ms":138150,"concrete_test":"Run a control condition with the same training samples and train/test split as Algorithm 1, but represent each training spectrogram by the VLM's raw visual embedding (e.g., last-layer patch embedding of the spectrogram image) instead of its text description; classify each test sample by cosine nearest-neighbor/centroid to the per-species embedding KB. Compare this to the 25.4% text-based accuracy. Also report the fixed-vs-progressive contrast with a permutation test over the 60 experiments (or 95% CIs). If raw-embedding retrieval matches/exceeds 25.4%, the claimed improvement is a supervised-retrieval effect, not evidence for natural-language knowledge; if it is significantly lower, the text bottleneck is confirmed.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"Alg. 1 builds KB(s) from VLM descriptions of labeled training spectrograms; Sec. 3.3 classifies a test spectrogram by TF-IDF cosine similarity of its VLM description against these per-species KB patterns. The vanilla VLM baseline (Sec. 3.1) is a zero-shot direct classification with no access to any training example. Hence the headline '92% improvement over vanilla VLMs' conflates the progressive-knowledge mechanism with the trivial effect of having labeled in-domain examples stored in a retrieval index. The correct control for the progressive component is the fixed-KB condition (18.5%, Table 1): the incremental gain is only 6.9 points (37% relative), and Sec. 2.4 defines H0/H1 but no significance test is reported anywhere. No experiment replaces the VLM text pattern with raw VLM/spectrogram embeddings of the same training samples, so it is unknown whether the natural-language library is the load-bearing carrier of signal. The paper's UMAP (Sec. 4.4) and the r=-0.44 quantity-accuracy correlation suggest the text bottleneck is real, but as reported the 92% number is equally consistent with 'any labeled training set improves retrieval,' which would not support the specific claim of knowledge-augmented VLMs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage, training-free pipeline for marine mammal classification from spectrograms. In Stage 1, a VLM generates natural-language descriptions of acoustic patterns in spectrograms; in Stage 2, these descriptions are matched against a knowledge base of per-species text patterns using TF-IDF cosine similarity. The knowledge base is initialized with synthetic expert patterns and grows via a progressive algorithm that filters VLM-generated patterns by undefined 'quality' and 'novelty' criteria. The authors report 25.4%±11.5% accuracy across 5-, 10-, and 24-species configurations, a 92% relative improvement over a vanilla VLM baseline (13.2%), and claim that the approach preserves interpretability and avoids retraining. The paper explicitly frames itself as preliminary and emphasizes expert-in-the-loop workflows over raw accuracy compared with specialized CNNs.","tokens_in":5354,"tokens_out":2810,"duration_ms":30353,"significance":"If the central claim were fully substantiated, the paper would make a modest but useful contribution: a no-retraining, interpretable route to bioacoustic classification that produces inspectable natural-language pattern descriptions. The authors are appropriately candid about limitations, include a UMAP-based semantic-gap analysis, and report performance on a public benchmark. The study also has a genuinely useful control condition, the Fixed Knowledge Base, even though it is not used to calibrate the headline claim. However, the reported '92% improvement' conflates the progressive knowledge mechanism with the trivial effect of giving the classifier access to labeled in-domain examples, and key components of Algorithm 1 are not specified. As it stands, the evidence supports a more modest claim: a retrieval index of labeled training descriptions improves zero-shot VLM accuracy, and adding a small, quality-filtered progressive set yields a further modest gain. The paper does not currently demonstrate that the natural-language knowledge representation is the load-bearing carrier of signal.","major_comments":[{"comment":"The headline '92% improvement over vanilla VLMs' conflates the progressive-knowledge mechanism with access to labeled training examples. The Vanilla VLM baseline (Section 3.1) has no access to any training sample, while the Progressive system stores VLM descriptions of labeled training spectrograms in its knowledge base (Algorithm 1). The correct control for the progressive component is the Fixed Knowledge Base condition reported in Table 1 (18.5%); on that comparison the incremental gain of the progressive system is only 6.9 points, or about 37% relative. The paper therefore overstates the contribution of progressive accumulation. In addition, no experiment replaces the VLM-generated text patterns with raw VLM or spectrogram embeddings of the same training samples. Without such a control, the results are equally consistent with 'any retrieval index of labeled examples improves classific","section":"§3.1, §4.2, Table 1"},{"comment":"The progressive learning algorithm is not reproducible as specified. Algorithm 1 lines 7–8 require evaluating quality(p) > θ_q and novelty(p, KB(s)) > θ_n, but the functions Q(p) and N(p) are never defined, and the thresholds θ_q and θ_n are never reported. The species-level aggregation weights 0.6, 0.3, and 0.1 in Section 3.3, and the TF-IDF n-gram range (1,3), are also presented without justification. If these parameters were tuned to maximize the reported 92% improvement, the headline result could be a tuning artifact. Please specify the filters, report the thresholds, and provide a sensitivity analysis or ablation showing how accuracy varies with these choices.","section":"§2.3, §3.2, Algorithm 1"},{"comment":"Section 2.4 formally defines H0 and H1 for the hypothesis that context improves accuracy, but no hypothesis test is reported anywhere in Section 4. The accuracy numbers in Table 2 include standard deviations (e.g., 31.8%±13.3% for 5 species), yet there is no statement of the number of independent runs, no paired comparison between conditions, and no confidence interval or p-value. Without this, the '92% improvement' could be within run-to-run variability. The paper should either report a proper statistical test over the 60 experiments or explicitly downgrade the claim to an exploratory observation.","section":"§2.4, §4.2, Table 2"},{"comment":"The UMAP analysis in Section 4.4 shows that VLM-generated pattern descriptions cluster by linguistic similarity rather than biological relationships. This is a direct challenge to the core premise of the two-stage design, namely that TF-IDF cosine similarity between text descriptions can separate species by preserving species-discriminative visual-acoustic information. The authors acknowledge this as a limitation, but the manuscript does not quantify how much discriminative information is retained. A quantitative analysis—for example, measuring the separability of per-species pattern sets, or showing that removing semantically ambiguous patterns changes accuracy—would help determine whether the semantic gap is a fixable engineering issue or a fundamental bottleneck. As written, the UMAP result is consistent with the possibility that the text bottleneck, not the progressive knowledge mech","section":"§4.4, Figure 3"}],"minor_comments":[{"comment":"The title and abstract describe 'knowledge-augmented Vision Language Models,' but the VLM itself is not modified; only the retrieval/classification stage is augmented. Consider rewording to 'knowledge-augmented VLM-based classification' to avoid implying model-level augmentation.","section":"Abstract / Title"},{"comment":"Typographical error: 'T rilemma' should be 'Trilemma.' Also, 'expert validation decisions are not capable' in Section 4.4 appears to be a missing-word or grammatical issue.","section":"§1, 'T rilemma'"},{"comment":"The Vanilla VLM baseline is listed as 12.5% in Table 1 but as 13.2% in Section 4.2. Please reconcile the numbers and state which value is used in the 92% calculation.","section":"Table 1"},{"comment":"The middle panel labels the process 'FixedKnowledge Base' without a space, and the bottom panel's 'QualityFilter' does not clearly correspond to the two separate quality and novelty filters in Algorithm 1. Minor labeling issue.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is explicitly labeled 'Work in Progress Contributions' and reports low absolute accuracy; I would not recommend it for publication in its current form without the additional controls and statistical analysis. The strongest useful component is the Fixed Knowledge Base comparison, which the authors should elevate in the revised framing. I have no concerns about citation integrity, but the paper would benefit from positioning relative to prior work on retrieval-augmented classification rather than presenting the 92% improvement as the main takeaway."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the take: this is a genuine work-in-progress, but the central claim is overstated. The 92% improvement over vanilla VLMs conflates having labeled examples in a retrieval index with the progressive-knowledge mechanism. The correct baseline is the fixed knowledge base, where the gain is about 6.9 points (37% relative), and no significance test is reported for even that. The stress-test note holds up on reading.\n\nWhat is actually new: applying progressive knowledge accumulation—VLM pattern extraction plus TF-IDF retrieval—to marine mammal spectrograms is a legitimate extension of known components. The paper also documents a useful failure mode: patterns cluster by linguistic similarity rather than biology, which explains the performance ceiling. The authors do some things right: time-based validation across 60 experiments, honest limitation statements, and a concrete expert-validation workflow.\n\nSoft spots: Algorithm 1's quality and novelty thresholds are never operationalized, and the 0.6/0.3/0.1 aggregation weights are presented without justification. The baseline inconsistency (13.2% vs 12.5%) is minor but should be fixed. Deeper, the paper never replaces the VLM text descriptions with raw VLM or spectrogram embeddings of the same training samples, so it is unknown whether natural language is the load-bearing carrier of signal or just any retrieval signal. The authors themselves flag the semantic gap and the negative quantity–accuracy correlation, so they are not hiding the problem.\n\nWho this is for: people working at the intersection of VLMs, bioacoustics, and interpretable classification. It would be a solid workshop submission; as a full paper it needs more rigor. I would send it to reviewers because the question is worth asking and the experiments are real, but I would expect major revisions and a request for code and data release. I would not cite the 92% claim in my own work; I might cite the semantic-gap observation with appropriate caveats.\n\nRecommendation: send to peer review, but be prepared to require proper controls, defined thresholds, and significance testing.","headline":"Honest feasibility study with a confounded headline: the 92% gain over vanilla VLMs mostly reflects labeled examples in the retrieval index, not progressive knowledge.","tokens_in":5789,"tokens_out":2802,"would_cite":false,"duration_ms":31354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that progressively accumulating VLM-generated pattern descriptions in a text knowledge base improves marine mammal spectrogram classification by 92% over a vanilla VLM, without retraining.","keywords":["marine mammal classification","bioacoustic spectrograms","vision language model","knowledge augmentation","progressive knowledge base","TF-IDF similarity","interpretability","zero-shot adaptation"],"falsifier":"If two acoustically distinct species whose VLM descriptions use the same phrases (e.g., 'burst sequences') are consistently confused even after many rounds of pattern accumulation, and removing generic phrases from the library does not change the confusion, then the text-similarity bottleneck is real and the central hypothesis fails in that setting.","tokens_in":4927,"feed_emoji":"🐋","tokens_out":4983,"duration_ms":47854,"temperature":0.7,"pith_summary":"The paper asks whether a vision-language model (VLM) can classify marine mammal calls from spectrograms without any model retraining, by accumulating a growing library of natural-language descriptions of acoustic patterns. It proposes a two-stage pipeline: a VLM writes a text description of each spectrogram, and a text-similarity engine matches that description against a per-species knowledge base built from expert-written rules plus VLM-generated patterns added over several rounds. Across 60 experiments on the Watkins database, the progressive knowledge base reached 25.4% mean accuracy versus 13.2% for a bare VLM, a 92% relative improvement, and it produced descriptions like \"20 Hz pulse trains with 12-second intervals\" that a biologist can check. The accuracy is far below specialized CNN models, but the paper's point is the trade-off: interpretable, no-retrain adaptation for rapid screening and expert validation. The paper also reports that the main bottleneck is the semantic gap: descriptions cluster by wording rather than by biological relationship, and generic patterns hurt accuracy as the knowledge base grows.","feed_headline":"VLM-written pattern notes lift whale-call ID by 92%","feed_subtitle":"Machine-written spectrogram descriptions lift accuracy 92% over a bare VLM, in text experts can check.","key_machinery":"The load-bearing mechanism is the progressive knowledge base combined with TF-IDF similarity matching. Each species has a text library of pattern descriptions; a query spectrogram is described by the VLM, vectorized with TF-IDF (n-gram range 1–3), and compared to library entries by cosine similarity. Species scores combine the best match, mean match, and a diversity term, so a single strong pattern description can carry classification while the library's breadth prevents overfitting to one phrase. The knowledge base evolves via a quality-and-novelty filter (thresholds θ and δ) that admits only new, non-generic patterns, with the authors reporting that 23% of extracted patterns were too gener","core_discovery":"The central claim is that a two-stage knowledge-augmented classification reframing—separating pattern extraction from species identification—lets a general-purpose VLM adapt to bioacoustic spectrograms without fine-tuning. In Stage 1 the VLM converts each spectrogram into a natural language pattern description; in Stage 2 a TF-IDF cosine similarity matcher scores that description against each species' knowledge base, and the species with the highest aggregated score (0.6 max, 0.3 mean, 0.1 diversity) is the prediction. The knowledge base starts with synthetic expert patterns and grows iteratively: patterns extracted from training samples are added only if they pass quality and novelty thresh","pith_inferences":["Because the paper's own UMAP analysis shows descriptions cluster by wording, a testable extension is to add a visual-anchor term to the similarity score (e.g., contrastive visual features) and measure whether cross-species linguistic confusions drop.","The reported negative correlation between pattern quantity and accuracy suggests a cheap lever: adaptive quality thresholds that reject near-duplicate generic patterns. Measuring accuracy against threshold θ directly would verify whether curation, not library size, is the binding constraint.","If the approach generalizes, the same two-stage text-knowledge design could be applied to other expert-visual domains (e.g., bird song spectrograms or medical imaging) where a VLM can describe patterns but not classify them directly."],"forward_implications":["If the central claim holds, marine mammal monitoring can bootstrap species screening from unlabeled recordings: a VLM writes descriptions, the knowledge base grows, and classification improves without retraining or manual annotation.","The natural-language output gives biologists a concrete validation workflow, e.g., checking a \"20 Hz pulse pattern with 12-second intervals\" prediction against known fin whale vocalization ranges before accepting it.","The approach is positioned as a rapid screening tool or annotation accelerator, not a replacement for specialized CNNs, so its practical value is in triage and data labeling rather than final autonomous decisions.","The engineering next steps the paper sketches—contrastive visual-linguistic alignment, hierarchical knowledge organization, uncertainty quantification, and RAG integration—are direct corollaries of the identified semantic-gap bottleneck."],"supporting_citations":[{"why":"Establishes that VLMs can act as few-shot spectrogram classifiers, the capability this paper builds on.","marker":"[8]"},{"why":"Supplies the retrieval-augmented generation approach adapted for progressive knowledge accumulation.","marker":"[9]"},{"why":"Provide the Watkins Marine Mammal Sound Database used in all experiments.","marker":"[12,13]"},{"why":"UMAP visualization used to expose the semantic gap between visual features and text descriptions.","marker":"[14]"},{"why":"NatureLM-audio, the foundation-model baseline compared for seen and unseen species.","marker":"[5]"}],"fun_headline_variants":["VLM descriptions boost whale-call ID by 92%, no fine-tuning","Two-stage VLM lifts bioacoustic accuracy by 92%","Pattern notes from VLM improve spectrogram ID 92%","Knowledge-augmented VLM beats baseline by 92% on whale calls","Stage-wise VLM reframing yields 92% accuracy gain"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The pipeline assumes that VLM-written text descriptions of spectrograms are similar for sounds of the same species and different for sounds of different species; if the text clusters by wording rather than by biology, the similarity-matching stage cannot separate species regardless of how many patterns accumulate.","fun_headline_variants_meta":{"raw":{"variants":["VLM descriptions boost whale-call ID by 92%, no fine-tuning","Two-stage VLM lifts bioacoustic accuracy by 92%","Pattern notes from VLM improve spectrogram ID 92%","Knowledge-augmented VLM beats baseline by 92% on whale calls","Stage-wise VLM reframing yields 92% accuracy gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000135,"raw_usage":{"total_tokens":897,"prompt_tokens":581,"completion_tokens":316,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":325,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":325,"tokens_out":316,"duration_ms":3796,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:07:20.288630+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If two acoustically distinct species whose VLM descriptions use the same phrases (e.g., 'burst sequences') are consistently confused even after many rounds of pattern accumulation, and removing generic phrases from the library does not change the confusion, then the text-similarity bottleneck is real and the central hypothesis fails in that setting.","supporting_citations":[{"cited_title":"Vision Language Models Are Few-Shot Audio Spectrogram Classifiers","cited_arxiv_id":"2411.12058","evidence_quote":"Establishes that VLMs can act as few-shot spectrogram classifiers, the capability this paper builds on."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-augmented generation approach adapted for progressive knowledge accumulation."},{"cited_title":"Hagge et al","cited_arxiv_id":null,"evidence_quote":"NatureLM-audio, the foundation-model baseline compared for seen and unseen species."}],"review_version":1}