{"id":"c5ea1189-2ce4-47c8-9dbb-b05291275a3e","arxiv_id":"2508.11187","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A contrastively trained speech-text model retrieves speech segments from free-form text descriptions of speaking style across 22 categories.","lead":"This paper trains speech and text encoders to match spoken audio with natural language descriptions of speaking style, enabling retrieval of speech by how it sounds rather than what it says. It reports retrieval results across 22 emotion and style categories.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation ranks by average similarity over all training paraphrases of the target style, so single-query free-form retrieval—the paper's central capability—is never measured, and the train/eval prompt overlap further hides generalization.","rationale":"The reader's weakest assumption (in-distribution prompt evaluation) is correct and important, but I sharpened it to a more direct methodological flaw: the evaluation protocol averages over all training-time paraphrases of the target style when computing the ranking. This means even the in-distribution numbers do not reflect the single-query usage scenario described in the abstract and Section III-D. A user will submit one arbitrary prompt; the system in the paper is given an ensemble of all label paraphrases. That is a stronger criticism than the reader's, because it affects the validity of the reported Recall@k even for queries that were seen during training. The two issues together make the headline 'strong retrieval performance' and the 'first open-ended setting' claim unsupported by the current experiments. I still believe the framework is plausible and the code release is valuable, so the paper is not fatally flawed; it needs a substantially revised evaluation before the central claim can be accepted. Since the reader already returned CONDITIONAL, my assessment does not change the verdict choice, but it strengthens the condition: the required revision should include single-prompt and held-out query evaluation, not just a caveat about generalization. I granted partial agreement because the reader did not explicitly identify the ensemble-averaging problem, which I consider the more load-bearing of the two issues.","tokens_in":10921,"tokens_out":5859,"duration_ms":60336,"concrete_test":"Build a held-out query set: for each of the 22 styles, collect 10 natural-language descriptions written by humans (or generated by a different LLM, e.g., Llama-3-70B) with instructions not to reuse any template or synonym from the released GitHub codebase. For each of the 1000 trials per style, use each held-out prompt individually as the sole query, rank the 401 candidates by cosine similarity, and compute Recall@1/5/10/20. Also, as a control, compute the same metrics using a single randomly sampled training prompt (not the average over all prompts) to isolate the effect of the ensemble averaging. If single-prompt Recall@1 drops by more than ~20% relative to Table II or if held-out prompts perform near chance, the central free-form generalization claim fails and the paper's headline numbers should be reinterpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-C evaluates retrieval by computing cosine similarity between each candidate speech embedding and every combination of the 11 templates × 5 synonyms for the target style, then ranking candidates by the average similarity across all of these prompts. This protocol has two load-bearing problems for the central 'open-ended, free-form query' claim. (1) It is an ensemble query over all ground-truth paraphrases, not a single user prompt; the system is effectively told 'here are all the ways to describe this style' and allowed to average. At inference (§III-D), a user submits one text prompt, and the reported Recall@k does not measure that scenario. (2) The prompt pool is the same one used during training (§IV-A2), so the evaluation is in-distribution. A novel query like 'sounds like they're talking over someone, annoyed' would be outside the 55 observed prompts, and the paper never tests such generalization. Because both issues inflate the numbers, the abstract's claim that the approach achieves 'strong retrieval performance' for arbitrary text queries is not supported by the reported tables. The ablations (Table III) show prompt augmentation matters, but they do not show the model can handle truly novel phrasings.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the task of expressive speech retrieval: retrieving speech utterances that match a natural-language description of their speaking style (e.g., \"sounds bored\" or \"whispered, sympathetic\"). The authors propose a CLAP-style framework with speech and text encoders trained via contrastive learning, an adversarial modality discriminator, an auxiliary style classification loss, and GPT-4o-generated prompt templates/synonyms for data augmentation. The system is evaluated on ESD and Expresso test sets, covering 22 unique style classes, with Recall@k against classification-based retrieval baselines, plus ablations showing the importance of each training component. The paper also analyzes performance across utterance durations and different encoder backbones.","tokens_in":11304,"tokens_out":5877,"duration_ms":70067,"significance":"If the headline claim were fully supported, this would be a useful new task and a reproducible baseline for cross-modal speech-style retrieval. The authors release code, data, and checkpoints, and the controlled trials with one positive vs. 400 distractors, together with systematic ablations, are strengths. The central weakness is that the evaluation does not actually measure the single-prompt, open-ended inference scenario described in the paper, and the prompts used in evaluation come from the same GPT-4o template/synonym pool used in training. The generalization claim is therefore not yet established. The framework and empirical inventory are nonetheless valuable and the issue is addressable by a revised evaluation protocol.","major_comments":[{"comment":"The evaluation protocol does not measure the inference scenario described in §III-D. In each trial, candidates are ranked by the average cosine similarity over all 11 templates × 5 synonyms for the target style—i.e., over all 55 ground-truth paraphrases. A real user submits one prompt, and a single-query retrieval accuracy is never reported. This ensemble-query protocol is likely to inflate Recall@k and cannot support the \"free-form,\" \"arbitrary text query\" claim. Please report per-query Recall@k by sampling one prompt per trial, or by averaging Recall@k over the individual prompts, and present these numbers in Table II.","section":"§IV-C, §III-D, Table II"},{"comment":"The statement that the best models are \"on par with, or even slightly exceeding, that of the classification-based baselines\" is contradicted at Recall@1. On Expresso, the best proposed model (BERT + emotion2vec) achieves R@1 = 0.8317 versus 0.9467 for the emotion2vec classifier; on ESD, the best proposed model (T5 + emotion2vec) achieves 0.6438 versus 0.6814. The R@5/R@10/R@20 numbers are close, but the R@1 gap is substantial. If the contribution is flexibility rather than parity, the comparison should be framed accordingly and should report the trade-off explicitly.","section":"§V-A, Table II"},{"comment":"The prompt-augmentation ablation reinforces the concern that the model is tuned to the GPT-4o template/synonym distribution. Removing prompt augmentation drops Expresso R@1 from 0.8159 to 0.5901 when evaluation uses the same diverse prompt pool. Since the same 55 prompts used in training are also used in evaluation, the paper never tests generalization to genuinely novel phrasings. Please add an evaluation on a separate, held-out prompt set (e.g., human-written descriptions or LLM paraphrases generated from a different seed/template family) and report single-query Recall@k for those prompts. This is necessary to support the paper's open-ended retrieval claim.","section":"§V-C, Table III"}],"minor_comments":[{"comment":"The set of 22 unique styles is obtained after excluding 7 Expresso categories (e.g., 'animal directed', 'child directed', 'narration'). The exclusion is reasonable, but the paper should explicitly list the excluded categories and note that the '22-style' claim depends on this filtering, since '22' is not simply the union of all annotated classes in the three datasets.","section":"§IV-A1"},{"comment":"It would help to state explicitly that each trial uses 55 prompt embeddings per style (11 templates × 5 synonyms) and that the final ranking is an ensemble average. The current wording is clear but does not give the reader the 55 number, which makes the protocol's nature easy to miss.","section":"§IV-C"},{"comment":"There is a minor typo in the caption: \"k = 1and k = 5\" should be \"k = 1 and k = 5.\" Also, the figure would benefit from confidence intervals or error bars, since the duration-binned results are based on a small number of test utterances.","section":"Fig. 3"},{"comment":"The row \"w/o aux. style classification loss\" is shown with dashes. The text explains that performance collapses, but it would be clearer to state in the table caption or a footnote that the model fails to learn a meaningful joint embedding, so no recall values are reported.","section":"Table III"},{"comment":"Reference [9] is described as the most closely related work, but it is not included as a direct baseline. The ablation table does include a \"w/o prompt augmentation\" condition that might serve a similar purpose, but a direct reimplementation or an explicit discussion of why [9] is not compared would strengthen the paper.","section":"§II-B"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core, but the central claim of open-ended, free-form query retrieval is not supported by the current evaluation protocol, which uses an ensemble over all training paraphrases and an in-distribution prompt pool. The required re-evaluation is substantial but well within the paper's scope, so I recommend major revision rather than rejection. I would also encourage the editor to request that the authors include confidence intervals in the main results, as the current tables report point estimates only."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the task framing: retrieving speech by natural-language descriptions of speaking style across 22 styles, well beyond the emotion-label setting of [9]. The paper also ships code, data, and checkpoints, which is real and should be counted in its favor. The ablations are informative—prompt augmentation matters, the auxiliary style loss is load-bearing, and the adversarial discriminator buys a clear chunk of R@1. Those are useful empirical findings for anyone working on speech-text alignment.\n\nNow the soft spots. The evaluation protocol in Section IV-C ranks candidates by average similarity over all 11 templates × 5 synonyms for the target style. That is an ensemble over the full ground-truth paraphrase set, not a single user query. The abstract says a user submits a free-form prompt; the reported Recall@k never measures that scenario. On top of that, the prompt pool is the same one used in training, so the generalization claim to arbitrary text queries is tested in-distribution. The stress-test note has this right: a novel phrasing like \"sounds like they're talking over someone, annoyed\" is simply not in the 55 observed prompts, and the paper gives no evidence for that case.\n\nSecond, the headline claim of \"on par with or slightly exceeding\" the classifier baselines is not what Table II shows. On Expresso, the emotion2vec classifier hits 0.9467 R@1 while the best retrieval model gets 0.8284. That is a real gap, and it matters because the classifier is restricted to fixed classes while the retrieval model is supposed to be open-ended—but the open-ended case is exactly what isn't measured. The authors should report single-query retrieval and held-out prompt generalization before making that comparison.\n\nThird, the closest prior work [9] is discussed but never directly compared, even though it is structurally similar. The paper explains the missing components, but a real comparison or a re-implementation of [9] would strengthen the novelty claim considerably.\n\nThese are not fatal flaws. The framework works, the task is new, and the paper is honest enough to show ablations that confirm the importance of its design choices. But the central claim about free-form, open-ended retrieval is currently unsupported, and the performance claim should be toned down. This deserves a serious referee: it is a solid empirical contribution that needs a revised evaluation protocol, not a desk reject.","headline":"New task framing and reproducible baselines for expressive speech retrieval, but the headline numbers are an ensemble-query, in-distribution estimate, not the single free-form query scenario the abstract promises.","tokens_in":11669,"tokens_out":1216,"would_cite":true,"duration_ms":16692,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces expressive speech retrieval, showing that speech and text encoders trained with contrastive and adversarial objectives can retrieve speech segments by natural-language style descriptions across 22 styles.","keywords":["expressive speech retrieval","spoken content retrieval","speaking style","cross-modal retrieval","contrastive learning","style representation learning","natural language prompts","prompt augmentation"],"falsifier":"Take the trained RoBERTa + emotion2vec model and prompt it with descriptions hand-written by humans that avoid the 11 generated templates and the generated synonym list. If Recall@1 on the same Expresso test items drops to near chance (about 1 in 401, the chance rate with 400 distractors) while performance on generated prompts stays high, the claim of open-ended generalization is falsified.","tokens_in":10870,"feed_emoji":"🎙️","tokens_out":5331,"duration_ms":55783,"temperature":0.7,"pith_summary":"The paper attempts to establish a new task: retrieving speech by how it is said, using open-ended natural language descriptions of emotion or speaking style as queries. It claims that a contrastively trained pair of speech and text encoders, aided by an adversarial modality alignment term, an auxiliary style classification loss, and LLM-based prompt augmentation, can embed audio and style descriptions into a shared space where cosine similarity ranks matching utterances accurately. Evidence is reported as Recall@k on three datasets spanning 22 styles, and the paper positions this as the first general, open-ended setting for expressive speech retrieval, beyond earlier fixed-emotion work. If correct, it would let users search large speech corpora with prompts like 'sarcastic' or 'calm and empathetic' without predefined labels.","feed_headline":"Search speech by speaking style, not just words","feed_subtitle":"New speech-text embeddings retrieve expressive audio from free-form style prompts across 22 emotion categories.","key_machinery":"The central object is a joint latent embedding space for speech and text, produced by a CLAP-style symmetric contrastive loss between paired speech segments and style prompts, an adversarial modality discriminator (with a gradient reversal layer) that makes the two modalities' embeddings harder to tell apart, and an auxiliary style classification loss on speech embeddings that the authors find critical to prevent retrieval collapse. Prompt augmentation with GPT-4o-generated templates and synonyms widens the query space. The mechanism's job is to make cosine similarity between a text query and speech embeddings a reliable ranking signal for speaking style.","core_discovery":"The paper's central claim is that how something is said can be made retrievable by what someone says about how it is said. It trains a speech encoder and a text encoder so that a speech segment and a free-form text description of its emotion or style land close together in a shared embedding space. At test time, a user prompt is embedded and compared by cosine similarity to cached speech embeddings; the top-ranked utterances are the retrieved matches. The authors report strong Recall@k on three datasets covering 22 emotion and style classes, and argue this is the first open-ended, general setting for expressive speech retrieval, going beyond previous work limited to a fixed set of emotion la","pith_inferences":["Because training and evaluation share the same GPT-4o template and synonym pool, the reported Recall@k is probably an in-distribution estimate; genuinely out-of-distribution user phrasings could show lower performance until prompt diversity is increased.","The same joint embedding could serve as a reusable style representation for neighboring tasks, such as zero-shot style classification or ranking style captions, not just retrieval.","A natural stress test is to evaluate on prompt phrasings written by humans independent of the generated pool, and to measure whether a similarity threshold can separate relevant from irrelevant styles without knowing the class labels.","The method's dependence on labeled style datasets suggests a next boundary: learning style-text alignment from weakly labeled or unlabeled expressive audio, where prompt augmentation and adversarial alignment may matter even more."],"forward_implications":["Retrieving expressive audio no longer requires a fixed label set: users can query with paraphrases and descriptive phrases, and the same index supports many open-ended styles.","Starting from a paralinguistics-specific speech model (emotion2vec) improves retrieval over a general-purpose model (WavLM), especially for nuanced styles on Expresso.","The language model chosen to encode prompts matters: RoBERTa and Flan-T5 consistently beat BERT and T5, even for short style descriptions.","Retrieval quality grows with target utterance duration; for utterances above roughly six seconds, top-5 retrieval is essentially perfect in the reported setting.","The auxiliary style classification loss is load-bearing: without it, the joint embedding space fails to learn and retrieval collapses."],"supporting_citations":[{"why":"Closest prior work on aligning emotion words with emotional audio; supplies the fixed-emotion baseline this work extends.","marker":"[9]"},{"why":"Provides the CLAP contrastive learning objective used to align speech and text embeddings.","marker":"[10]"},{"why":"Supplies the IEMOCAP emotional speech dataset used for training.","marker":"[11]"},{"why":"Supplies the ESD emotional speech dataset and its standard test split.","marker":"[12]"},{"why":"Supplies the Expresso expressive speech corpus and its test split, including broader speaking styles.","marker":"[13]"},{"why":"Supplies the WavLM speech encoder backbone compared in the experiments.","marker":"[31]"},{"why":"Supplies the emotion2vec speech encoder backbone, which gives the best retrieval performance.","marker":"[32]"},{"why":"Supplies the GPT-4o generated prompt templates and synonyms used for prompt augmentation.","marker":"[40]"},{"why":"Supplies the domain-adversarial training method used for the modality discriminator.","marker":"[38]"},{"why":"Supplies the gradient reversal layer that enables the adversarial loss on the encoders.","marker":"[39]"}],"fun_headline_variants":["How it's said becomes searchable via text","Expressive speech retrieval via style text","Match audio to free-form descriptions of speaking style","Search spoken audio with natural language style prompts"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The open-ended generalization claim rests on the assumption that the GPT-4o-generated prompt templates and synonyms used in training are representative of arbitrary user queries, because the evaluation prompts are drawn from the same pool.","fun_headline_variants_meta":{"raw":{"variants":["How it's said becomes searchable via text","Expressive speech retrieval via style text","Match audio to free-form descriptions of speaking style","Search spoken audio with natural language style prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00034,"raw_usage":{"total_tokens":1663,"prompt_tokens":649,"completion_tokens":1014,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":393,"completion_tokens_details":{"reasoning_tokens":958}},"tokens_in":393,"tokens_out":1014,"duration_ms":11274,"temperature":1.0,"reasoning_tokens":958,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:03:51.663074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained RoBERTa + emotion2vec model and prompt it with descriptions hand-written by humans that avoid the 11 generated templates and the generated synonym list. If Recall@1 on the same Expresso test items drops to near chance (about 1 in 401, the chance rate with 400 distractors) while performance on generated prompts stays high, the claim of open-ended generalization is falsified.","supporting_citations":[{"cited_title":"Prompting audios using acoustic properties for emotion representation,","cited_arxiv_id":null,"evidence_quote":"Closest prior work on aligning emotion words with emotional audio; supplies the fixed-emotion baseline this work extends."},{"cited_title":"Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,","cited_arxiv_id":null,"evidence_quote":"Provides the CLAP contrastive learning objective used to align speech and text embeddings."},{"cited_title":"IEMOCAP: Interactive emotional dyadic motion capture database,","cited_arxiv_id":null,"evidence_quote":"Supplies the IEMOCAP emotional speech dataset used for training."},{"cited_title":"Emotional voice conversion: Theory, databases and ESD,","cited_arxiv_id":null,"evidence_quote":"Supplies the ESD emotional speech dataset and its standard test split."},{"cited_title":"Expresso: A benchmark and analysis of discrete expressive speech resynthesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the Expresso expressive speech corpus and its test split, including broader speaking styles."},{"cited_title":"emo- tion2vec: Self-supervised pre-training for speech emotion representation,","cited_arxiv_id":null,"evidence_quote":"Supplies the emotion2vec speech encoder backbone, which gives the best retrieval performance."},{"cited_title":"Domain-adversarial training of neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the domain-adversarial training method used for the modality discriminator."},{"cited_title":"Unsupervised domain adaptation by backpropagation,","cited_arxiv_id":null,"evidence_quote":"Supplies the gradient reversal layer that enables the adversarial loss on the encoders."}],"review_version":1}