{"id":"7458f998-2e79-4eec-b8fd-bc931e8ce35c","arxiv_id":"2411.12058","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GPT-4o classifies environmental sounds from spectrogram images in a few-shot setting with 59% cross-validated accuracy on ESC-10, beating a commercial audio language model and roughly matching human experts.","lead":"This paper shows that GPT-4o can classify audio recordings from spectrogram images, reaching 59% accuracy on the ESC-10 dataset with a few labeled examples. It proposes spectrogram classification as a new challenge for vision language models and compares them with human experts and commercial audio models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pretraining contamination is the load-bearing threat: if GPT-4o memorized ESC-10 spectrogram-label pairs, the reported 59% accuracy reflects retrieval, not spectrogram reasoning, and the paper performs no contamination check despite acknowledging VLMs have seen spectrograms in pretraining.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the absence of a contamination check on GPT-4o's pretraining data. The central claim is not merely that GPT-4o scores 59% on ESC-10, but that this score demonstrates a capability to classify audio from spectrogram images. If the score is substantially driven by memorization of ESC-10, the capability claim collapses. The paper provides no evidence against this: it cites that VLMs have seen spectrograms during pretraining, and ESC-10 is a standard public benchmark that predates GPT-4o's training cutoff. The paper does include a limitation admitting VLMs do not match traditional audio classifiers and that performance degrades on 50 classes, but neither addresses contamination. A private-dataset replication is the decisive test because it removes the possibility of overlap with training data. I credit the paper for cross-validating on ESC-10 and for providing the tuned-versus-default comparison, but these do not mitigate the contamination threat. The tuning-on-fold-1 issue is real but secondary: it inflates the reported number and the Gemini comparison, whereas contamination challenges the interpretation of what the model is doing. Thus the reader's conditional verdict is appropriate: the paper should be accepted only if the authors can demonstrate non-memorized generalization, for example via a private held-out set. Since this was already the reader's recommendation, no verdict adjustment is needed.","tokens_in":7221,"tokens_out":5434,"duration_ms":57032,"concrete_test":"Run the identical few-shot protocol (same prompt template, same K-means example selection, same spectrogram parameters) on 80 newly recorded or privately held audio clips spanning the same 10 ESC-10 classes, with no public web presence before the test date. If GPT-4o accuracy on this private set is at or near chance (roughly 10-20%), the ESC-10 results are likely explained by pretraining contamination. If accuracy remains substantially above chance (e.g., >40%), the model is genuinely extracting class-relevant information from novel spectrograms, and the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VLMs can classify audio by reasoning over spectrogram images, supported by GPT-4o's 59.00% cross-validated accuracy on ESC-10 and its superiority over Gemini-1.5 audio (49.62%). For this claim to hold, GPT-4o's accuracy must come from generalizing visual time-frequency patterns, not from recognizing specific ESC-10 spectrograms memorized during pretraining. The paper states in Section 1 that VLMs 'have seen spectrograms and associated text during pre-training,' and ESC-10 is a public, widely used dataset that very likely appears in web-scale training corpora. Yet no contamination check is performed. If GPT-4o memorized ESC-10 image-label associations, the cross-validated accuracy is an artifact of retrieval: the model could match a test spectrogram to a memorized training image rather than extract acoustic evidence from the spectrogram. The 14% accuracy on the ESC-50 subset, while suggestive of poor generalization, does not rule out partial memorization of ESC-10. This concern is compounded by the fact that hyperparameters and few-shot examples were selected on the first fold and then included in the cross-validated average, inflating the headline number; however, that inflation would weaken the numeric claim, whereas contamination would undermine the qualitative interpretation of the result as 'visual spectrogram classification.' The central claim therefore rests on the unverified assumption that the model is not reciting memorized training data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Visual Spectrogram Classification (VSC), a task in which vision-language models classify audio content from spectrogram images, and evaluates six commercial VLMs in zero-shot and few-shot settings on ESC-10. The authors report that GPT-4o achieves 59.00% cross-validated accuracy on ESC-10, outperforming Gemini-1.5 Pro's audio-based accuracy of 49.62%, and that GPT-4o slightly exceeds ensembled human expert accuracy on the first fold (73.75% vs. 72.50%). The paper includes ablations over spectrogram representations and few-shot example-selection strategies, and it proposes VSC as a benchmark and as a route toward audio caption augmentation.","tokens_in":7565,"tokens_out":6100,"duration_ms":56970,"significance":"If the central result survives a contamination check, it would be a notable empirical demonstration that VLMs can extract acoustic content from spectrogram images and compete with an audio-native commercial model. The paper's strengths are its breadth (six VLMs, three human experts, one audio language model), the ablation over spectrogram hyperparameters and few-shot example selection, and the explicit limitations section. The proposed VSC task is reasonable, and the human-expert data are potentially useful. However, the current evidence does not establish that the reported accuracy comes from visual reasoning rather than from memorized training associations, and the statistical support for the headline comparisons is thin.","major_comments":[{"comment":"The central interpretation assumes GPT-4o classifies spectrograms by visual reasoning rather than by retrieving memorized ESC-10 image-label associations. The authors state in Section 1 that VLMs 'have seen spectrograms and associated text during pre-training,' and ESC-10 is a public dataset from 2015 that could plausibly appear in web-scale training corpora, but no contamination check is reported. The ESC-50 subset result (14%) does not rule out partial memorization of ESC-10. I would need at least one control experiment—for example, classifying spectrograms from newly recorded or private audio of the same classes, classifying frequency-inverted or scrambled versions of ESC-10 spectrograms, or comparing against a simple image-retrieval baseline—before accepting the qualitative claim that the accuracy reflects visual spectrogram understanding.","section":"§1, §3.5, §4"},{"comment":"The headline 59.00% cross-validated accuracy is produced by a protocol whose hyperparameters and example-selection rules were chosen on the first fold, yet the paper reports no per-fold accuracies, standard errors, or confidence intervals. For example, Table 2 shows 76.25% for '2 per class K-means Mel' on fold 1, but the cross-validated section does not state whether this same configuration was applied to all folds or whether the 59.00% is an average over single runs. The manuscript should specify the exact protocol used for each fold, report all five fold accuracies, and provide a significance test (e.g., a paired test against Gemini-1.5 audio or a binomial confidence interval on the pooled accuracy). Without this, the 59.00% versus 49.62% gap is not statistically grounded.","section":"§3.3, §3.5, Table 2"},{"comment":"The VLM-versus-ALM comparison is under-specified. Table 1 lists Gemini-1.5 Flash/Pro (Audio) accuracies, but the appendix only documents the VSC prompts, not the audio prompts, the number of audio examples, the exact ESC-10 folds, or the API settings used for the audio-conditioned models. If the comparison is meant to support the abstract's '59.00% vs. 49.62%' claim, the audio classification protocol must be described in the same detail as the VSC protocol, including whether the same few-shot examples and folds were used. Otherwise the gap could be due to prompt design or evaluation differences rather than to the modality.","section":"§3.5, Table 1"},{"comment":"The claim that GPT-4o 'even perform[s] slightly better than human experts' is based on three experts and a single 80-item fold, where the ensembled human accuracy (72.5%) and GPT-4o (73.75%) differ by one correct response. No confidence interval or chance-level baseline for the human task is reported. This statement should be either softened to 'in this small comparison' or supported by additional expert evaluators and/or more folds.","section":"§3.4, Abstract"}],"minor_comments":[{"comment":"The terms '10-shot', '20-shot', and '30-shot' refer to total examples (1, 2, or 3 per class), which is potentially confusing; define the notation at first use.","section":"§2.3, §3.3"},{"comment":"Please report exact model versions (e.g., gpt-4o-2024-05-13), API call parameters, temperature, and any sampling settings, since commercial APIs can change over time and the results are otherwise not reproducible.","section":"§3.1, Appendix B"},{"comment":"The confusion matrices are small and difficult to read; a larger figure or a table of per-class precision and recall would make the error analysis more useful.","section":"Figure 3"},{"comment":"The statement that VLMs will improve as they 'potentially encounter more spectrograms in their training data' inadvertently underscores the contamination concern; this should be addressed explicitly rather than treated only as a scaling prediction.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short benchmark-style empirical report. Its main risk is not novelty but evidential standards: the missing contamination control and the lack of statistical reporting are both fixable within the scope of a revision. I would not recommend rejection, but the authors should either add the controls or substantially soften the interpretation of the central claim. If the journal requires methodological novelty or released artifacts, the absence of code and exact prompts may be a scope concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine capability demonstration with a load-bearing contamination problem. The paper shows GPT-4o can label ESC-10 spectrograms with 59% cross-validated accuracy when given a few examples, and that it beats Gemini-1.5's audio-input classifier on the same data. That result is new, as far as I know, and the authors did real work on spectrogram representation and example selection. They also ran a small human-expert comparison and came out slightly ahead of the ensembled experts on the first fold. The paper is honest about its main limitation: VLMs are still far behind conventional audio classifiers.\n\nWhat's actually soft: the paper never addresses pretraining contamination, even though it admits in Section 1 that VLMs have seen spectrograms and text during pretraining. ESC-10 is a standard public dataset, so the strong chance is that GPT-4o has memorized these exact spectrogram-label pairs. The 59% could be retrieval rather than reasoning. The stress-test note is right that this undermines the qualitative interpretation, not just the number. A simple contamination probe (e.g., classifying held-out audio from a new dataset, or scrambling the spectrograms) would settle this.\n\nSecond, the evaluation protocol is partly fitted. The authors tuned spectrogram hyperparameters, example counts, and K-means settings on the first fold, then report cross-validated accuracy that includes that fold. That's not a clean cross-validation, and it could inflate the headline number. They also give no error bars or significance tests; the 59% vs 49.62% gap looks real but is not quantified.\n\nThird, the comparison set is thin: only one commercial audio language model, no traditional audio classifier baselines, and three human experts. The experts are a nice touch, but with n=3 the comparison is anecdotal.\n\nThat said, the central claim is not contradicted by the paper's own data, and the fixes are straightforward. The paper deserves a serious referee. I'd send it to peer review with a request for a contamination check, a cleaner protocol, and baseline comparisons. The audience is people working on VLM capabilities and audio benchmarking; it's a useful datapoint, not a breakthrough. I'd bring it to a reading group if we have one on multimodal evaluation, but I wouldn't cite it in my own work until the contamination question is resolved.","headline":"A real capability result that is currently undercut by an unaddressed contamination risk and a fitted evaluation protocol.","tokens_in":8065,"tokens_out":2317,"would_cite":false,"duration_ms":22744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision language models classify environmental sounds by reading spectrogram images, with GPT-4o reaching 59% on ESC-10.","keywords":["vision language models","spectrogram classification","few-shot learning","ESC-10","environmental sound classification","in-context learning","audio language models","visual spectrogram classification"],"falsifier":"Take a set of environmental sounds that were recorded and published after GPT-4o's training data was collected, render them into spectrograms using the paper's exact pipeline, and run the same 10-shot prompt; if accuracy falls to near chance while accuracy on the original ESC-10 folds stays near 59%, the reported performance is largely memorization rather than spectrogram reasoning.","tokens_in":7007,"feed_emoji":"🔊","tokens_out":5820,"duration_ms":55837,"temperature":0.7,"pith_summary":"The paper tries to establish that vision language models (VLMs) can act as few-shot audio classifiers when audio is presented as spectrogram images, without any audio-native input. On the ESC-10 environmental sound dataset, GPT-4o reaches 59.00% cross-validated accuracy after tuning the spectrogram rendering and choosing few-shot examples by clustering; the same setup beats the commercial audio-language model Gemini-1.5 on the equivalent audio task (49.62%) and slightly exceeds ensembled human expert performance on one fold (73.75% vs. 72.50%). The authors propose visual spectrogram classification (VSC) as a new task and a benchmark for VLMs, and point toward using spectrograms to ground audio captions. A reader should care because it suggests visual pattern recognition alone can extract meaningful audio content, and because it reframes audio understanding as an image understanding problem.","feed_headline":"GPT-4o classifies sounds from spectrogram images, hitting 59%","feed_subtitle":"GPT-4o hits 59% on ESC-10 with a few examples, beating Gemini's audio model and matching experts.","key_machinery":"The load-bearing mechanism is the pairing of a deliberately rendered spectrogram image with in-context few-shot prompting. Spectrograms are computed with a 2,048-sample window and 512-sample hop, resampled to 22,050 Hz, log-scaled in frequency and amplitude, and colored with viridis; the paper then shows the model example spectrograms for each class inside the prompt and asks it to name the class of a new image. Example selection matters: choosing spectrograms closest to K-means cluster centroids of mel spectrograms outperforms random or hand-picked examples, and two examples per class gives the best first-fold accuracy (76.25%).","core_discovery":"On its own terms, the paper's central discovery is that a VLM prompted with a spectrogram and a handful of labeled example spectrograms can classify environmental sounds at a level comparable to a commercial audio-native model and to human experts. The strongest result is GPT-4o's 59.00% cross-validated accuracy on ESC-10, obtained by searching spectrogram hyperparameters (logarithmic frequency and amplitude, viridis colormap, axis labels, no colorbar) and by selecting few-shot examples near K-means cluster centroids of mel spectrograms. The paper also shows that few-shot examples help far more than zero-shot prompts, and that performance collapses when the class count grows, with 14% accuracy on a 50-class ESC-50 subset.","pith_inferences":["If the result survives a contamination check, it would imply that spectrograms are a sufficiently universal visual language that pretraining on natural images transfers to time-frequency plots; a testable extension would be to render novel sounds recorded after the model's training cutoff and measure whether few-shot accuracy persists.","The same prompting recipe could be applied to non-environmental audio, such as music tagging or speech emotion classification, and the clustering-based example selection may transfer as a general strategy for choosing in-context exemplars.","The authors' comparison with Gemini-1.5 is not a matched comparison, since Gemini receives raw audio while GPT-4o receives images, so the headline numbers should be read as relative capabilities of two different input modalities, not as a direct model race."],"forward_implications":["If the central claim holds, audio classification can be performed by a model that never hears audio, only sees it, making audio tasks available to vision-language pipelines.","Visual spectrogram classification becomes a reusable few-shot benchmark for comparing VLM visual reasoning across models.","Spectrogram-plus-language prompting could be used to augment or ground audio captions, addressing vague or hallucinated captions in captioning datasets.","The sharp accuracy drop from 10 to 50 classes warns that current VLMs are only practical for small-class sound discrimination, not large-scale taxonomy.","The near-human expert result on the first fold suggests VSC is a meaningful test of human-level auditory pattern reading, not a trivial image task."],"supporting_citations":[{"why":"Provides the ESC-10 and ESC-50 datasets and fold structure used for all classification evaluations.","marker":"[14]"},{"why":"Supplies the in-context few-shot learning methodology that the paper adapts to spectrogram images.","marker":"[6]"},{"why":"Documents the GPT-4o model that produces the paper's main few-shot VSC results.","marker":"[1]"},{"why":"Introduces Gemini-1.5, the commercial audio-language baseline compared against GPT-4o on the equivalent audio task.","marker":"[17]"},{"why":"Represents current audio language models whose architectural limitations motivate using VLMs with spectrograms.","marker":"[5]"},{"why":"Defines the audio captioning dataset that motivates the proposed use case of grounding captions with spectrograms.","marker":"[7]"}],"fun_headline_variants":["Vision models turn spectrograms into sound classifiers, hitting 59% on ESC-10","GPT-4o classifies spectrogram images, beating Gemini's audio model on ESC-10","Few-shot spectrogram prompts let GPT-4o match experts on ESC-10","VLMs can classify audio from spectrograms: GPT-4o hits 59% on ESC-10","Spectrogram-based few-shot classification: GPT-4o beats Gemini's audio model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4o's correct answers come from actually reading the spectrogram patterns, not from having memorized ESC-10 spectrogram images or their class labels during pretraining, and the paper performs no contamination check to rule that out.","fun_headline_variants_meta":{"raw":{"variants":["Vision models turn spectrograms into sound classifiers, hitting 59% on ESC-10","GPT-4o classifies spectrogram images, beating Gemini's audio model on ESC-10","Few-shot spectrogram prompts let GPT-4o match experts on ESC-10","VLMs can classify audio from spectrograms: GPT-4o hits 59% on ESC-10","Spectrogram-based few-shot classification: GPT-4o beats Gemini's audio model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2956,"prompt_tokens":902,"completion_tokens":2054,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1937}},"tokens_in":518,"tokens_out":2054,"duration_ms":14548,"temperature":1.0,"reasoning_tokens":1937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:58:01.901999+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of environmental sounds that were recorded and published after GPT-4o's training data was collected, render them into spectrograms using the paper's exact pipeline, and run the same 10-shot prompt; if accuracy falls to near chance while accuracy on the original ESC-10 folds stays near 59%, the reported performance is largely memorization rather than spectrogram reasoning.","supporting_citations":[{"cited_title":"Pengi: An audio language model for audio tasks, 2024","cited_arxiv_id":null,"evidence_quote":"Represents current audio language models whose architectural limitations motivate using VLMs with spectrograms."},{"cited_title":"Clotho: an audio captioning dataset","cited_arxiv_id":null,"evidence_quote":"Defines the audio captioning dataset that motivates the proposed use case of grounding captions with spectrograms."}],"review_version":1}