Pith. sign in

REVIEW 4 major objections 6 minor 60 references

CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing fixed class templates with per-audio enriched prompts fixes homograph and auditory-illusion errors in audio-to-image generation, the paper reports.

desk verdict A sensible prompt-enrichment pipeline with consistent gains on three datasets, but the headline AIS numbers may be inflated by CLAP being both the selector and the metric. read the letter →

arxiv 2507.18750 v1 pith:GWPCHKSL submitted 2025-07-24 cs.MM cs.SDeess.AS

Hyunwoo Oh , SeungJu Cha , Kwanyoung Lee , Si-Woo Kim , Dong-Jin Kim This is my paper · ORCID
classification cs.MMcs.SDeess.AS
keywords audio-to-imagegenerationsemanticmisalignmentpromptenrichmenthomographdisambiguationauditoryillusionencoderadaptationcross-modalretrievaldiffusionmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CatchPhrase is a training recipe for audio-to-image generation that attacks the semantic poverty of the labels rather than the encoder alignment itself. Its claim is that fixed template prompts such as "a photo of X" systematically mislead the generator on homographs ("crane" the bird versus "crane" the machine) and on sounds whose class label underdescribes the instance ("engine idling," "drilling"). The framework mines candidate prompts from an LLM using visual, auditory, and semantic queries plus zero-shot audio captions, then selects the best prompt per audio clip with a similarity-based filter-and-retrieve step. Trained with those selected prompts as anchors, a lightweight adapter improves audio-image similarity over the template-based baseline on US8K, ESC-50, and VGGSound while also lowering FID and KID.

What carries the argument

The load-bearing object is the EXPrompt, an enriched cross-modal prompt that fuses text knowledge about a class (from an LLM asked to describe how the class looks, sounds, and what it means) with audio-derived captions produced by a zero-shot audio captioning model. The EXPrompt Selector then applies two score functions in the shared audio-text embedding space: a filtering score (Eq. 4) that rewards prompts similar to same-class audio and penalizes similarity to audio from other classes, keeping top-K prompts per class; and a retrieval score (Eq. 5) that selects the single most similar prompt for each audio clip. The selected prompts supervise a compact mapping network that adapts a frozen audio encoder's features into the text-encoder space of a pretrained text-to-image model, trained with InfoNCE contrastive loss combined with MSE, reconstruction, and adversarial losses.

What would settle it

On a held-out set of audio clips from classes with known homographs and ambiguous instances, ask human raters which candidate prompt best matches each clip; if a substantial fraction of the time raters prefer a prompt that the CLAP-based selector ranked below the top few, then the selection mechanism is not picking the semantically right anchors and the reported alignment gains rest on a faulty oracle.

Watch

Extended reading notes

Core claim

The central claim is that the limiting factor in audio-to-image generation is the text prompt derived from a weak class label, not the cross-modal encoder itself. CatchPhrase shows that enriching labels with LLM-generated visual, auditory, and semantic descriptions together with zero-shot audio captions, then filtering those candidates against same-class versus other-class audio and retrieving the top match for each audio instance, yields prompts that resolve both class-level homograph confusion and instance-level auditory illusion. With these prompts as training targets, a lightweight mapping network inserted between a frozen audio encoder and a frozen text encoder produces images with higher Audio-Image Similarity (a score measuring how well the generated image matches the input audio) than GlueGen on US8K (0.191 vs 0.144), ESC-50 (0.242 vs 0.192), and VGGSound (0.202 vs 0.195), and lower FID (65.62 vs 76.66) and KID (0.0119 vs 0.0188) on VGGSound.

Load-bearing premise

The whole filter-and-retrieve pipeline trusts that the audio-text similarity scores used to select prompts reflect true semantic agreement; if those scores are unreliable for these sounds, the chosen prompts will be noisy and the trained adapter will inherit the error.

Editorial extensions

If this is right

  • Template-based adapters leave systematic class- and instance-level misalignment; replacing templates with selected enriched prompts raises AIS on all three reported datasets.
  • Homograph errors such as "siren" and auditory-illusion errors such as "engine idling" are reduced, and the supplementary retrieval results show EXPrompts are far more likely to be the top-1 text match for an audio clip than template prompts.
  • Auditory-only or onomatopoeia-heavy prompts degrade generation, so the semantic query from the LLM is doing essential disambiguation work, not decoration.
  • Only a lightweight mapping network is trained, so the approach plugs into any frozen text-to-image model without retraining the generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same CLAP space is used both to select prompts and to compute the reported AIS metric, the gains may partly reflect optimizing the selection toward the evaluation metric; a held-out human preference test or a different embedding model for selection would separate method improvement from metric alignment.
  • The EXPrompt plus selector recipe transfers in principle to any label-poor generation task where one modality should disambiguate another, such as generating images from animal vocalizations or industrial machinery sounds, since the pipeline only requires weak labels plus an audio-text embedder.
  • A testable variant would swap the CLAP embedder for a different audio-text model at selection time and measure AIS; if the improvement shrinks, the contribution is tied to the embedding space rather than to prompt enrichment per se.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. CatchPhrase proposes a framework for audio-to-image generation that mitigates semantic misalignment caused by homographs and auditory illusions. The method first mines enriched textual prompts (EXPrompts) from weak class labels using LLMs and audio captioning models, then applies a CLAP-based filtering and retrieval procedure to select one semantically aligned prompt per audio instance, and finally trains a lightweight mapping network that adapts a frozen text-to-image diffusion model to audio features, using MSE, reconstruction, adversarial, and InfoNCE losses. Experiments on UrbanSound8K, ESC50, and VGGSound report higher Audio-Image Similarity (AIS) than Sound2Scene, GlueGen, and SonicDiffusion, together with improved FID/KID, plus ablations showing the contribution of each component. The paper claims that the enriched prompts and the prompt selector reduce both class-level and instance-level misalignment.

Significance. If the reported gains are robust, CatchPhrase would be a practically useful contribution to audio-to-image generation: it addresses a real failure mode (weak labels in audio classification datasets causing homograph and auditory-illusion errors), it is built from modular and reproducible components, and it includes ablations, a retrieval-based sanity check, and a small user study. The method's reliance on CLAP both as the prompt-selection oracle and as the basis for the evaluation metric is, however, a serious validation concern, and the absence of uncertainty quantification makes the small reported differences difficult to interpret. The central idea is defensible, but the paper currently does not establish that the AIS gains reflect genuine image-semantic alignment rather than prompt-side similarity.

major comments (4)
  1. [§3.3, §4.1, Eq. (4)-(5)] The evaluation may be partly circular: the same cross-modal similarity paradigm (CLAP-style audio-text embeddings) is used both to filter and retrieve training prompts in Eq. (4)-(5) and to compute the headline AIS metric in Section 4.1. The paper does not state whether the AIS checkpoint is the same CLAP model used in the selector, nor does it provide an independent (human or external model) validation of semantic alignment. Since the training objective directly moves mapped audio features toward text features of CLAP-selected prompts, a model that merely matches those prompt embeddings could inflate AIS without improving true image semantics. Please report the exact AIS computation, disclose the checkpoint, and add an evaluation that does not rely on the same embedding space used for prompt selection, e.g., human judgment or a separately trained audio-image alignment model.
  2. [§4.2, Table 1, §4.1 'random seed is fixed at 42'] All quantitative claims are made without error bars, multiple seeds, or significance tests. The headline differences are often small (e.g., VGGSound AIS 0.2017 vs 0.1950 for GlueGen), and the InfoNCE ablation in Table 6 reports differences of 0.0005-0.0007 on AIS. With a single fixed seed, it is impossible to determine whether any of these differences are reliable. Please run at least 3-5 seeds and report mean and standard deviation, or otherwise justify that the differences are above the noise floor.
  3. [Supplementary Table 9 and Section 4.3 'Component Sequence Ablation Studies'] The filtering hyperparameters NAS and top-K are swept over the values shown in Supplementary Table 9 and the best values are selected using AIS, which is the same metric used for the main comparisons. Since no held-out validation split or nested selection procedure is described, the reported numbers are likely optimistically biased. Please specify the validation protocol used for hyperparameter selection and report test performance under that protocol.
  4. [§3.4, Table 6] Table 6 shows that InfoNCE contributes only 0.0005 (US8K), 0.0001 (ESC50), and 0.0007 (VGGSound) to AIS. Given the lack of uncertainty quantification, the text's claim that the InfoNCE loss 'contributes to mitigating auditory illusions' (Section 4.3) is not supported by the presented evidence. If this loss is retained as a core contribution, please provide a more sensitive evaluation (e.g., instance-level human judgments on auditory-illusion examples) or soften the claim.
minor comments (6)
  1. [Throughout] There are several typos and formatting issues: 'adpater' (Figure 3 and text), 'auido' (Table 7 caption), 'confusion cased by homographs' (Section 1), and inconsistent spacing around equations and in Table 6. A careful proofread is needed.
  2. [§3.3, Eq. (4)] The notation for the filtering score is confusing: the variable r_p seems to be both a single prompt embedding and an element of a set, and the indices i, j are not fully defined beyond 'I is the set of audio samples with the same class label as the prompt'. Please clarify the indexing and the exact aggregation over the NAS audio samples.
  3. [§3.4, Eq. (7)] The loss weights are described only as 'determined empirically'. Please report the range considered and the selection criterion, or at least state that the weights were fixed before evaluating the main comparisons.
  4. [§4.1 'Evaluation Metrics'] The paper says AIS follows [51], but it is not described precisely enough to be reproducible: which audio encoder, which image encoder, and what aggregation over generated images are used? Please provide the exact AIS computation or a reference to an implementation.
  5. [Supplementary, Table 13] The audio-to-text retrieval results (R@1) are reported as raw percentages, but the comparison is only between CatchPhrase prompts and a single template set; the retrieval experiment does not control for the number of candidate prompts or the effect of prompt diversity. Please state the candidate pool size and how ties are handled.
  6. [Supplementary, Section G] The user study is small (30 participants, 15 pairs) and the statistical significance of the 64.22% vs 35.78% split is not reported. Please add a confidence interval or a significance test, and specify whether the participants were shown the audio, the image, and the text in both conditions consistently.

Circularity Check

1 steps flagged · score 4.0 of 10

A supporting validation is circular: EXPrompts are selected by CLAP and then 'validated' by CLAP audio-to-text retrieval, while the main AIS metric is left undefined, so the central alignment claim is not fully independent of the selection oracle.

  1. fitted input called prediction [Supplementary Materials, Section F (Audio to Text Retrieval)]
    "To evaluate whether EXPrompt is truly related to the audio, we compared it against the predefined template-based prompts used in the original GlueGen. Specifically, we measured whether the prompt with the highest retrieval top-1 similarity to the test set audio samples originated from EXPrompt or from the baseline prompts. For this evaluation, we employed CLAP as the text and audio encoder."

    The EXPrompt Selector (Section 3.3, Eqs. 4-5) already filters and retrieves prompts by CLAP cosine similarity to class audio. This supplementary 'validity' check then measures CLAP audio-to-text retrieval on test audio and reports that EXPrompts are retrieved more often than GlueGen templates. Because the test criterion is the same CLAP model used to choose the prompts, the high retrieval rate is a restatement of the selection rule: prompts were selected to be CLAP-nearest to the class and training audio, so they are CLAP-nearest to same-class test audio if CLAP generalizes. The experiment confirms that the selector did what it was designed to do, but it does not independently establish that CLAP similarity corresponds to true semantic alignment.

full rationale

The main derivation is not a pure self-citation or definitional tautology: EXPrompt Mining uses LLMs and audio captioning models, and the mapping network is trained with MSE, reconstruction, adversarial, and InfoNCE objectives against CLIP text features of selected prompts. The reported FID/KID gains and the user study in the supplementary are external evidence that the generated images are semantically relevant, and no load-bearing claim rests on the authors' own prior work. The one definite circularity is in the supplementary audio-to-text retrieval validation, which uses CLAP both as the prompt-selection oracle and as the evaluation oracle, making the 'EXPrompt is truly related to the audio' result forced by construction rather than independently tested. The paper also leaves the AIS metric undefined, so the main alignment score cannot be checked against the selection oracle; if AIS uses the same CLAP/CLIP-style embeddings as Eqs. 4-5, the central AIS gains would be partially circular. Because the central claim retains independent image-quality and human-evaluation support, a moderate score of 4 is appropriate rather than a higher score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method depends on several tuned hyperparameters and domain assumptions, primarily the reliability of CLAP as a cross-modal similarity oracle and the validity of the AIS evaluation metric. No new physical entities are introduced. The free parameters are the loss weights, contrastive temperature, number of negatives, and filtering choices.

free parameters (4)
  • Loss weights lambda_1, lambda_2, lambda_3, lambda_4 = 1.0, 10000, 10000, 0.5
    Set empirically in Section 4.1 and used in Eq. 7; no sensitivity analysis is provided.
  • InfoNCE temperature tau = 0.8
    Chosen in Section 4.1 for the contrastive loss in Eq. 6.
  • Number of InfoNCE negatives N = 8
    Set in Section 4.1; the effect of N is not studied.
  • Filtering subset size NAS and top-K = NAS=10, top-K=10 for main experiments
    Selected based on AIS sweeps in Supplementary Section D, Table 9; the chosen values are tuned on the evaluation metric.
assumptions (4)
  • domain assumption CLAP similarity scores reliably indicate semantic alignment between audio and text prompts.
    Used in Eq. 4 and 5 for filtering and retrieval; if CLAP is unreliable, selected prompts will be wrong and the adapter will be misaligned.
  • domain assumption CLIP text encoder features are an appropriate target space for aligning audio features.
    The MSE loss (Eq. 1) trains the adapter to map audio features to the CLIP text embedding of the selected prompt; this assumes text features are the right semantic carrier for image generation.
  • domain assumption The candidate EXPrompts generated by GPT-3.5 and EnCLAP cover the semantic variation within each class.
    EXPrompt Mining assumes that the candidate pool contains a prompt appropriate for each audio instance; otherwise filtering and retrieval cannot recover good targets.
  • domain assumption The AIS metric measures audio-image semantic alignment.
    AIS is adopted from AudioToken without independent validation; if AIS is biased, the reported improvements are partly an artifact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation." pith.science (2026). https://pith.science/paper/GWPCHKSL

@misc{pith2026250718750,
  author       = {Pith},
  title        = {Pith review of: CatchPhrase: EXPrompt-Guided Encoder Adaptation for Audio-to-Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GWPCHKSL}},
  note         = {Machine review of arXiv:2507.18750}
}
read the original abstract

We propose CatchPhrase, a novel audio-to-image generation framework designed to mitigate semantic misalignment between audio inputs and generated images. While recent advances in multi-modal encoders have enabled progress in cross-modal generation, ambiguity stemming from homographs and auditory illusions continues to hinder accurate alignment. To address this issue, CatchPhrase generates enriched cross-modal semantic prompts (EXPrompt Mining) from weak class labels by leveraging large language models (LLMs) and audio captioning models (ACMs). To address both class-level and instance-level misalignment, we apply multi-modal filtering and retrieval to select the most semantically aligned prompt for each audio sample (EXPrompt Selector). A lightweight mapping network is then trained to adapt pre-trained text-to-image generation models to audio input. Extensive experiments on multiple audio classification datasets demonstrate that CatchPhrase improves audio-to-image alignment and consistently enhances generation quality by mitigating semantic misalignment.

Figures

Figures reproduced from arXiv: 2507.18750 by the authors.

Figure 1
Figure 1. Visual demonstration of CatchPhrase. Unlike prior method [35] relying on fixed templates for audio-text alignment, CatchPhrase uses semantically enriched prompts from audio and text cues, producing well-aligned results for each audio sample. Abstract We propose CatchPhrase, a novel audio-to-image generation frame￾work designed to mitigate semantic misalignment between audio inputs and generated images. While recent … view at source ↗
Figure 2
Figure 2. Examples of cross-modal misalignment. When the prompts “crane” and “bat” are used for image generation, the model often confuses their meanings, resulting in class-level misalignment (i.e., homographs). Similarly, given audio samples such as “engine idling” or “drilling,” the model produces instance-level misaligned results due to auditory illusions. Although GlueGen [35] showed notable performance in audio￾to-image… view at source ↗
Figure 3
Figure 3. Pipeline of each stage in CatchPhrase. (Left) EXPrompts are generated by leveraging class-relevant knowledge from LLMs and ACMs. To pair each audio sample with the most semantically appropriate prompt, we introduce the EXPrompt Selector, which enhances both class-level and instance-level alignment during adpater training. (Right) Illustration of the inference-time pipeline. LLMs ACMs (Visual) Describe what a {Dog ba… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Overview of EXPrompt Mining and EXPrompt. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Comparison of image results between previous works and CatchPhrase, demonstrating enhanced semantic alignment. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative results from the CatchPhrase compo [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Impact of EXPrompts on image generation com [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Generated images with using audio queries in text [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Various component sequences in consisting CatchPhrase. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Differences in generated images based on the top-K [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 22 canonical work pages

  1. [1]

    Abdelrahman Abdelhamed, Mahmoud Afifi, and Alec Go. 2024. What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models. arXiv preprint arXiv:2405.15668 (2024)

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Burak Can Biner, Farrin Marouf Sofian, Umur Berkay Karakaş, Duygu Cey- lan, Erkut Erdem, and Aykut Erdem. 2024. SonicDiffusion: Audio-Driven Im- age Generation and Editing with Pretrained Diffusion Models. arXiv preprint arXiv:2405.00878 (2024)

    arXiv 2024

  4. [4]

    Sutherland, Michael Arbel, and Arthur Gretton

    Mikołaj Bińkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton

  5. [5]

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18392–18402

    2023

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  7. [7]

    SeungJu Cha, Kwanyoung Lee, Ye-Chan Kim, Hyunwoo Oh, and Dong-Jin Kim

    work page

  8. [8]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. Vg- gsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 721–725

    2020

Show all 60 references
  1. [9]

    Mia Chiquier, Utkarsh Mall, and Carl Vondrick. 2025. Evolving interpretable visual classifiers with large language models. InEuropean Conference on Computer Vision. Springer, 183–201

    2025

  2. [10]

    Chengbin Du, Yanxi Li, Zhongwei Qiu, and Chang Xu. 2024. Stable diffusion is unstable. Advances in Neural Information Processing Systems 36 (2024)

    2024

  3. [11]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang

  4. [12]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 15180–15190

    2023

  5. [13]

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 976–980

    2022

  6. [14]

    Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2024. Optimizing prompts for text-to-image generation. Advances in Neural Information Processing Systems 36 (2024)

    2024

  7. [15]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2018. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. arXiv:1706.08500 [cs.LG] https://arxiv.org/abs/1706. 08500

    2018 arXiv

  8. [16]

    Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo

  9. [17]

    Jaeyeon Kim, Jaeyoon Jung, Jinjoo Lee, and Sang Hoon Woo. 2024. Enclap: Combining neural audio codec and audio-text joint embedding for automated audio captioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 6735–6739

    2024

  10. [18]

    Si-Woo Kim, MinJu Jeon, Ye-Chan Kim, Soeun Lee, Taewhan Kim, and Dong-Jin Kim. 2025. SynC: Synthetic Image Caption Dataset Refinement with One-to- many Mapping for Zero-shot Image Captioning. In Proceedings of the 33rd ACM International Conference on Multimedia

    2025

  11. [19]

    Ye-Chan Kim, SeungJu Cha, Si-Woo Kim, Taewhan Kim, and Dong-Jin Kim. 2025. SIDA: Synthetic Image Driven Zero-shot Domain Adaptation. In Proceedings of the 33rd ACM International Conference on Multimedia

    2025

  12. [20]

    Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun Kwak. 2023. AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion. arXiv:2305.04001 [cs.CV] https://arxiv.org/abs/2305.04001

    2023 arXiv

  13. [21]

    Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chanyoung Kim, Jinkyu Kim, and Sangpil Kim. 2022. Sound-guided semantic image manipu- lation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3377–3386

    2022

  14. [22]

    Taegyeong Lee, Jeonghun Kang, Hyeonyu Kim, and Taehwan Kim. 2023. Gen- erating Realistic Images from In-the-wild Sounds. arXiv:2309.02405 [cs.CV] https://arxiv.org/abs/2309.02405

    2023 arXiv

  15. [23]

    Rongjie Li, Yu Wu, and Xuming He. 2024. Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13428–13437

    2024

  16. [24]

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22511–22521

    2023

  17. [25]

    Harry McGurk and John MacDonald. 1976. Hearing lips and seeing voices.Nature 264, 5588 (1976), 746–748

    1976

  18. [26]

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt- assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, a...

    2024

  19. [27]

    Sachit Menon and Carl Vondrick. 2022. Visual Classification via Description from Large Language Models. arXiv:2210.07183 [cs.CV] https://arxiv.org/abs/2210. 07183

    2022 arXiv

  20. [28]

    Jackson Michaels, Juncheng B Li, Laura Yao, Lijun Yu, Zach Wood-Doughty, and Florian Metze. 2024. Audio-Journey: Open Domain Latent Diffusion Based Text-To-Audio Generation. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . ...

    2024

  21. [29]

    Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. 2023. Verbs in Action: Improving verb understanding in video- language models. arXiv:2304.06708 [cs.CV] https://arxiv.org/abs/2304.06708

    2023 arXiv

  22. [30]

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)

    2021 arXiv

  23. [31]

    Yuta Oshima, Masahiro Suzuki, Yutaka Matsuo, and Hiroki Furuta. 2025. Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search. arXiv:2501.19252 [cs.CV] https://arxiv.org/abs/2501.19252

    2025

  24. [32]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Christiano, Jan Leike,...

    2022 arXiv

  25. [33]

    Karol J Piczak. 2015. ESC: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia . 1015–1018

    2015

  26. [34]

    Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. 2023. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 15691– 15701

    2023

  27. [35]

    Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. 2023. Gluegen: Plug and play multi-modal encoders for x-to-image generation. InProceedings of the IEEE/CVF International Conference on Computer Vision. 23085–23096

    2023

  28. [36]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  29. [37]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

    2018

  30. [38]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9

    2019

  31. [39]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

  32. [40]

    Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. 2023. Linguistic binding in diffusion models: Enhancing at- tribute correspondence through attention map alignment. Advances in Neural Information Processing Systems 36 (2023), 3536–3559

    2023

  33. [41]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695

    2022

  34. [42]

    Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, and Ron Kimmel. 2023. FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions. arXiv preprint arXiv:2305.17718 (2023)

    2023 arXiv

  35. [43]

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural infor...

    2022

  36. [44]

    arXiv preprint arXiv:2204.06125 1, 2 (2022), 3

    Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2 (2022), 3

    2022 arXiv

  37. [45]

    Kim Sung-Bin, Kim Jun-Seong, Junseok Ko, Yewon Kim, and Tae-Hyun Oh. 2024. SoundBrush: Sound as a Brush for Visual Scene Editing. arXiv:2501.00645 [cs.CV] https://arxiv.org/abs/2501.00645

    2024 arXiv

  38. [46]

    Kim Sung-Bin, Arda Senocak, Hyunwoo Ha, Andrew Owens, and Tae-Hyun Oh

  39. [47]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_ alpaca

    2023

  40. [48]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    2018 arXiv

  41. [49]

    Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia. 1041–1044

    2014

  42. [50]

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech an...

    2023

  43. [51]

    Guy Yariv, Itai Gat, Lior Wolf, Yossi Adi, and Idan Schwartz. 2023. AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation. arXiv:2305.13050 [cs.SD] https://arxiv.org/abs/2305.13050

    2023 arXiv

  44. [52]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Sound to visual scene generation by audio-to-visual latent alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6430–6440

  45. [53]

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional con- trol to text-to-image diffusion models. InProceedings of the IEEE/CVF International Conference on Computer Vision . 3836–3847

    2023

  46. [54]

    dog” and “engine idling

    Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. 2023. Remodiffuse: Retrieval-augmented motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 364–373. 10 Supplementary Materi...

    2023

  47. [55]

    Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. 2022. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 4563–4567

    2022

  48. [58]

    Rushikesh Zawar, Shaurya Dewan, Prakanshul Saxena, Yingshan Chang, Andrew Luo, and Yonatan Bisk. 2024. DiffusionPID: Interpreting Diffusion via Partial Information Decomposition. arXiv:2406.05191 [cs.CV] https://arxiv.org/abs/2406. 05191

    2024 arXiv

  49. [2021]

    arXiv:1801.01401 [stat.ML] https://arxiv.org/ abs/1801.01401

    Demystifying MMD GANs. arXiv:1801.01401 [stat.ML] https://arxiv.org/ abs/1801.01401

    arXiv

  50. [2022]

    arXiv preprint arXiv:2211.09699 (2022)

    Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699 (2022)

    2022 arXiv

  51. [2023]

    InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2023

  52. [2025]

    arXiv:2503.16406 [cs.GR] https://arxiv.org/abs/2503.16406

    VerbDiff: Text-Only Diffusion Models with Enhanced Interaction Aware- ness. arXiv:2503.16406 [cs.GR] https://arxiv.org/abs/2503.16406

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.