Pith. sign in

REVIEW 5 major objections 7 minor 57 references

Moodifier: MLLM-Enhanced Emotion-Driven Image Editing

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Moodifier turns abstract emotions into targeted, structure-preserving image edits.

desk verdict A genuinely useful new dataset and a sensible editing pipeline, but the headline claim about content preservation is not supported by Table 4. read the letter →

arxiv 2507.14024 v1 pith:GYMU6O3P submitted 2025-07-18 cs.CV

Jiarong Ye , Sharon X. Huang This is my paper · ORCID
classification cs.CV
keywords emotion-drivenimageeditingaffectivecomputingvision-languagemodelCLIPfine-tuningdiffusionattentioncontrolmultimodallargelanguageemotiondatasetzero-shotclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that abstract emotions can be turned into concrete, localized image edits by combining a large emotion-annotated dataset, a CLIP model fine-tuned on it, and a training-free diffusion editing pipeline. The authors introduce MoodArchive, a dataset of over 8 million images with hierarchical emotion captions generated by LLaVA-NeXT and partially checked by human evaluators. On top of it they train MoodifyCLIP, which maps an emotion name to detailed visual attributes, and then build Moodifier, which uses a multimodal large language model to write emotion-specific prompts and attention masks that tell a diffusion model which regions to change. The paper reports that Moodifier outperforms existing editing methods in both emotional accuracy and content preservation across faces, fashion, jewelry, and home decor. For a general reader, the point is that mood may become a first-class editing parameter, like color or style.

What carries the argument

The load-bearing mechanism is cross-attention control in a text-conditioned diffusion model. Cross-attention maps are the matrices that decide how strongly each text token influences each image region. Moodifier takes the source image, has LLaVA-NeXT produce an emotion-specific prompt and a spatial emotion-stimulus mask, then denoises the image's latent representation while blending the cross-attention maps of the source prompt and the target prompt: inside the mask it uses the target attention, outside it keeps the source attention, with a schedule that fixes structure early and adds emotional detail late. MoodifyCLIP supplies the emotion-tuned text encoder that turns the multimodal LLM's prompt into embeddings the diffusion model and the CLIP-based similarity measure can use.

What would settle it

Sample several thousand never-validated MoodArchive images and have independent annotators judge whether the overall emotion and the three emotional stimuli match the image. If agreement with the LLaVA-NeXT annotations is close to chance or much lower than the 85% seen in the validated 10k subset, then MoodifyCLIP is trained on systematically wrong labels and Moodifier's edits inherit that error.

Watch

Extended reading notes

Core claim

The paper's central claim is that emotion-driven editing works best when the emotion is resolved into two concrete things before the diffusion model runs: a textual description of how the target emotion should look in this specific image, and a spatial mask saying where that change belongs. MoodifyCLIP contributes the first by learning emotion-to-attribute mappings from MoodArchive's hierarchical captions, and LLaVA-NeXT contributes the mask and prompt at edit time. Moodifier then performs the edit by inverting the source image into a latent and running the diffusion steps with cross-attention maps blended inside and outside the mask. The paper reports that this combined design beats both prompt-only and mask-only ablations, and that it outperforms existing editing methods in emotional accuracy while keeping structural metrics competitive.

Load-bearing premise

The whole pipeline assumes the automatically generated emotion captions in MoodArchive are mostly correct, although only about 10,000 of the 8 million-plus images were checked by human annotators.

Editorial extensions

If this is right

  • With Moodifier working as described, an animation artist can generate an emotional sequence for a character while keeping the character's identity fixed, which is exactly the use case the paper highlights.
  • Because the editing stage is training-free, adding a new emotion or a new object category does not require retraining the editor; only the multimodal LLM instruction changes.
  • MoodifyCLIP's zero-shot gains imply the same backbone could serve as an emotion-understanding layer for other vision-language tasks, not only for editing.
  • The ablation pattern, where prompts alone weaken structure and masks alone weaken emotion, supports the paper's design principle that emotional editing needs guidance on both what to change and how to change it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the same two-step recipe, an emotion-tuned encoder plus attention-mask-controlled diffusion, could be applied to style transfer, where the role of 'emotion' is played by a named aesthetic or period style.
  • A testable extension: decouple emotional semantics from raw caption length by fine-tuning a control model on non-emotional long captions matched for length; if it matches MoodifyCLIP, the reported gain could be about length rather than emotion.
  • The paper's 85% human-approval statistic is measured on a random 10k subset; because the authors state hallucinations are inevitable at the 8M scale, a stratified validation sample over the remaining images would give a more trustworthy error bound for downstream use.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents Moodifier, an emotion-driven image editing framework built from three components: MoodArchive, an 8M+ image dataset with hierarchical emotion annotations generated by LLaVA and partially validated by human evaluators; MoodifyCLIP, a CLIP variant fine-tuned on MoodArchive for emotion understanding; and Moodifier, a training-free editing method that uses an MLLM to produce emotion-specific prompts and spatial masks, then applies attention-controlled diffusion to modify only emotion-relevant regions. The authors report that MoodifyCLIP improves zero-shot emotion classification and retrieval over CLIP baselines (Tables 1–2), and that Moodifier achieves the highest emotional-accuracy scores and human preference among ten editing baselines (Table 4). The paper claims that the system outperforms existing methods in both emotional accuracy and content preservation, a claim the evidence only partially supports.

Significance. If the results hold, the paper makes a useful contribution to affective computing and image editing: it provides a large-scale emotional image dataset with hierarchical captions, a fine-tuned CLIP model that transfers to held-out emotion datasets (Emotion6, EmoSet, Emotic, SentiCap, Affection), and a training-free editing pipeline that produces strong emotional transformations while retaining competitive structural fidelity. The evaluation design has strengths: MoodifyCLIP is tested on external transfer datasets, the editing CLIP score uses a standard pretrained CLIP rather than MoodifyCLIP, and a 10k-image human validation study is reported. The main weakness is that the headline claim of superiority in content preservation is not consistent with Table 4, and several evaluation-protocol details (prompt adaptation, MTurk preference design, absence of variance reporting) need clarification or revision. With a more cautious framing and additional statistical analysis, the work would be a solid candidate for publication; in its current form the central claim overstates the evidence.

major comments (5)
  1. [Abstract and §4.2, Table 4] The abstract's claim that Moodifier 'outperforms existing methods in both emotional accuracy and content preservation' is not supported by Table 4. Moodifier is best only on LPIPS (94.9) and CLIP similarity (16.13); Inversion-Free Editing attains higher PSNR (24.11 vs 22.19), lower MSE (80.1 vs 127.3), higher SSIM (82.9 vs 82.2), and lower structure distance (14.6 vs 17.4), and Plug-and-Play has lower structure distance (13.9). No significance tests or per-image variance are reported for these editing metrics, so the evidence supports a claim of superior emotional accuracy (CLIP score and human preference) with competitive, not superior, preservation. The headline claim should be tempered or supported by appropriate statistical tests.
  2. [§4.1, Table 4 caption, §4.2] The evaluation protocol does not establish a fair comparison. The caption states that 'Emotions were adapted to category-appropriate attributes for each domain,' but the main text does not specify whether the baselines received the same MLLM-generated detailed prompts and masks as Moodifier. If baselines were given only generic emotion words while Moodifier received richly contextualized prompts, the emotional-accuracy advantage could partly reflect prompt quality rather than the editing method. In addition, the MTurk protocol asks participants to select their 'top 3 most-preferred models' overall, which conflates emotional appeal with content preservation; separate ratings of content preservation and emotional accuracy would be needed to validate the two-pronged claim.
  3. [§4.1–4.2, Table 4] No measure of variability or statistical significance is reported for any editing metric in Table 4, despite the paper's claims of 'consistently outperforming' and of achieving an 'optimal balance.' This is conspicuous because Tables 1–2 report standard deviations. The editing evaluation uses only 30 images per category across 8 categories and 27 emotions, so per-condition sample sizes are small. Confidence intervals, per-image distributions, or significance tests should be reported before drawing conclusions about superiority on any dimension.
  4. [§3.1–3.2, Eqs. (2)–(4)] The dataset annotation pipeline is validated on only 10k of 8M+ images (0.125%), and Section 3.2 concedes that hallucinations are 'inevitable at this scale.' The paper does not measure the error rate on the unvalidated portion or show that MoodifyCLIP's gains are robust to annotation noise. If LLaVA systematically mislabels emotional stimuli for certain object classes, the fine-grained loss and the optimal-transport loss would learn incorrect associations, and the downstream editing pipeline (Algorithm 1) would inherit those errors. An analysis of robustness—for instance, training on varying fractions of human-validated captions or injecting synthetic noise into captions—would strengthen the claims about MoodArchive and MoodifyCLIP.
  5. [§3.3, Algorithm 1, Fig. 3] The mechanism for generating the emotion stimulus masks ME is underspecified. Algorithm 1 and the text state that the MLLM 'generates' both the prompt PE and the attention maps ME, but Fig. 3 includes a 'Grounded SAM' component in the pipeline, and the text never explains how an MLLM produces spatial attention maps (e.g., via coordinate prompting, segmentation grounding, or attention from the MLLM's internal layers). This ambiguity prevents replication; the authors should specify the exact procedure for obtaining ME, including whether it involves additional models and what prompt is used.
minor comments (7)
  1. [§3.2, Eq. (2)] The denominator in the Lfg_v2t term contains a malformed expression: 'exp(T f g i,j, cos(I f g i,m)/τ)' should presumably read 'exp(cos(T f g i,j, I f g i,m)/τ)'. Please also check the notation consistency of subscripts in Eqs. (1) and (2).
  2. [§3.3, Algorithm 1] Line 3, 'PE ← MoodifyCLIP(PE)' is unclear: MoodifyCLIP is a vision-language alignment model, not a text-rewriting module. Please specify the operation being performed (e.g., selecting emotion-relevant tokens, reweighting text embeddings, or generating a refined prompt).
  3. [§4.2, Table 4] The sentence 'Methods focusing on pixel-level fidelity (Plug-and-Play, Inversion-Free Editing) struggle with emotional expression' is too strong given that Inversion-Free Editing attains the highest CLIP score among all non-Moodifier methods (14.23) and the best baseline human Emotion Accuracy (47.9). Rephrase to reflect that these methods are relatively weaker on emotional accuracy while often stronger on fidelity.
  4. [Table 5] The bold/† markings in Table 5 appear inconsistent. For example, the MoodifyCLIP-ViT-L-14 row has PSNR 21.55 and LPIPS 102.2, both column maxima, yet neither is bolded or marked †, while the same row has numbers marked † that are not second-best. Please recheck the table formatting against the note.
  5. [§4.1, Table 4] The list of baselines includes 'Direct Inversion,' which is likely intended to correspond to PnP Inversion (Ref. [58]), but the reference list and table do not make this mapping explicit. Clarify which method each row implements.
  6. [Algorithm 1, line 11] The indicator '1_ME>0' is not defined. Please define the notation and clarify how the thresholded mask relates to the attention map ME and to the latent-space blending described in the text.
  7. [§4.1] The paper states that the 240 test images are 'from MoodArchive, no overlap with MoodifyCLIP training set,' but it does not explain how this split is guaranteed given that MoodArchive comprises the training data. Specify the split mechanism (e.g., the held-out MoodArchive-5k subset).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoodifyCLIP is tested on external datasets, the editing CLIP score uses a pretrained external CLIP model, and no load-bearing claim reduces to its own inputs by construction.

full rationale

The paper's derivation chain is self-contained against external evidence rather than circular. MoodifyCLIP is evaluated zero-shot on Emotion6, EmoSet, and Emotic for classification and on SentiCap and Affection for retrieval, all of which are independent of MoodArchive; the only in-distribution test set, MoodArchive-5k, is held out from training. The editing evaluation's CLIP similarity score is explicitly computed with a pretrained OpenAI CLIP-ViT-Large-Patch14, not with MoodifyCLIP, so the metric is not identical to the model being proposed. The editing method itself builds on externally published techniques (Prompt-to-Prompt attention control, Inversion-Free Editing, LLaVA-NeXT) and does not rely on any self-citation, uniqueness theorem, or author-imported ansatz to justify its design. The acknowledged limitations—that only 10k of 8M+ MoodArchive images were human-validated, that LLaVA hallucinations are 'inevitable at this scale,' and that no standardized benchmark exists for emotion-driven editing—are explicitly stated and affect generalization confidence, not circularity. The mismatch between the abstract claim of superiority in content preservation and Table 4, where Moodifier is best on LPIPS but not on PSNR, MSE, or SSIM, is a correctness and calibration concern, not a circular-reasoning concern, because the reported numbers are external measurements rather than constructions of the method's own definitions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on (a) the reliability of LLaVA-generated annotations at 8M scale, (b) a small number of unstated hyperparameters, and (c) the assumption that CLIP similarity reflects emotional accuracy. These are not independently verified to the level the headline claims require.

free parameters (3)
  • Loss weights λf, λs, λfg, λot = Not reported
    Weights for the four loss terms in Eq. 5; presumably tuned on a validation split, but values are not given in the main text.
  • Attention control threshold τc = Not reported
    Controls when refined attention is used (Eq. 6); hand-chosen hyperparameter, value not reported.
  • Sinkhorn entropy regularization ε = Not reported
    Regularization in optimal transport loss (Eq. 4); hyperparameter not specified in the main text.
assumptions (4)
  • domain assumption LLaVA-NeXT-generated structured captions accurately capture emotional content and stimuli in images
    MoodArchive is built entirely from automatic annotations; human validation covers only 10k of 8M+ images and 15% of those were rejected. If annotations are systematically wrong, MoodifyCLIP learns incorrect emotion-visual mappings.
  • ad hoc to paper Three local emotional stimuli per image are sufficient to represent the emotional triggers
    The caption template fixes exactly three stimuli (Fig. 2), a structural choice not derived from theory or data.
  • domain assumption Emotions can be transferred across domains by modifying localized visual attributes while preserving identity
    The editing algorithm relies on the premise that the MLLM-generated attention maps identify the correct regions to change and that textual emotion prompts can drive appropriate visual transformations.
  • domain assumption Standard CLIP similarity is a valid proxy for emotional accuracy in edited images
    The main editing metric is CLIP score between the output and target emotion text; if CLIP does not capture emotional nuance, the quantitative evaluation of emotional accuracy is weak.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Moodifier: MLLM-Enhanced Emotion-Driven Image Editing." pith.science (2026). https://pith.science/paper/GYMU6O3P

@misc{pith2026250714024,
  author       = {Pith},
  title        = {Pith review of: Moodifier: MLLM-Enhanced Emotion-Driven Image Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GYMU6O3P}},
  note         = {Machine review of arXiv:2507.14024}
}
read the original abstract

Bridging emotions and visual content for emotion-driven image editing holds great potential in creative industries, yet precise manipulation remains challenging due to the abstract nature of emotions and their varied manifestations across different contexts. We tackle this challenge with an integrated approach consisting of three complementary components. First, we introduce MoodArchive, an 8M+ image dataset with detailed hierarchical emotional annotations generated by LLaVA and partially validated by human evaluators. Second, we develop MoodifyCLIP, a vision-language model fine-tuned on MoodArchive to translate abstract emotions into specific visual attributes. Third, we propose Moodifier, a training-free editing model leveraging MoodifyCLIP and multimodal large language models (MLLMs) to enable precise emotional transformations while preserving content integrity. Our system works across diverse domains such as character expressions, fashion design, jewelry, and home d\'ecor, enabling creators to quickly visualize emotional variations while preserving identity and structure. Extensive experimental evaluations show that Moodifier outperforms existing methods in both emotional accuracy and content preservation, providing contextually appropriate edits. By linking abstract emotions to concrete visual changes, our solution unlocks new possibilities for emotional content creation in real-world applications. We will release the MoodArchive dataset, MoodifyCLIP model, and make the Moodifier code and demo publicly available upon acceptance.

Figures

Figures reproduced from arXiv: 2507.14024 by the authors.

Figure 1
Figure 1. Our emotion-driven Moodifier image editing system performs consistently across diverse scenarios such as product mockups, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Each image in MoodArchive is paired with a structured [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Our MLLM-enhanced Moodifier (right panel (3)) integrates ideas from instruction-based and attention control approaches, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons with other editing methods. Zoom in for better view. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: A qualitative example for ablation. Ours allows faith [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 47 canonical work pages

  1. [1]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3813–3824, 2023. 1, 6

    work page 2023

  2. [2]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- structpix2pix: Learning to follow image editing instructions. 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 18392–18402, 2022. 3

    work page 2023

  3. [3]

    Instructdiffusion: A generalist mod- eling interface for vision tasks

    Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, and Baining Guo. Instructdiffusion: A generalist mod- eling interface for vision tasks. In Proc. CVPR, 2024. 1, 3, 6

    work page 2024

  4. [4]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 1

    arXiv 2015

  5. [5]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 1

    work page 2014

  6. [6]

    Universals and cultural differences in the judgments of facial expressions of emotion

    Paul Ekman, Wallace V Friesen, Maureen O’sullivan, An- thony Chan, Irene Diacoyanni-Tarlatzis, Karl Heider, Rainer Krause, William Ayhan LeCompte, Tom Pitcairn, Pio E Ricci-Bitti, et al. Universals and cultural differences in the judgments of facial expressions of emotion. Journal of per- sonality and social psychology, 53(4):712, 1987. 2

    work page 1987

  7. [7]

    Affective image classi- fication using features inspired by psychology and art theory

    Jana Machajdik and Allan Hanbury. Affective image classi- fication using features inspired by psychology and art theory. In Proceedings of the 18th ACM international conference on Multimedia, pages 83–92, 2010. 2, 3

    work page 2010

  8. [8]

    A mixed bag of emotions: Model, pre- dict, and transfer emotion distributions

    Kuan-Chuan Peng, Tsuhan Chen, Amir Sadovnik, and An- drew C Gallagher. A mixed bag of emotions: Model, pre- dict, and transfer emotion distributions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 860–868, 2015. 5

    work page 2015

Show all 57 references
  1. [9]

    Emoset: A large-scale visual emotion dataset with rich attributes

    Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischin- ski, Danny Cohen-Or, and Hui Huang. Emoset: A large-scale visual emotion dataset with rich attributes. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 20383–20394, 2023. 5

    2023

  2. [10]

    Context based emotion recognition using emotic dataset

    Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. Context based emotion recognition using emotic dataset. IEEE transactions on pattern analysis and machine intelligence, 42(11):2755–2766, 2019. 5

    2019

  3. [11]

    Affection: Learning affective explanations for real-world visual data

    Panos Achlioptas, Maks Ovsjanikov, Leonidas Guibas, and Sergey Tulyakov. Affection: Learning affective explanations for real-world visual data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6641–6651, 2023. 2, 3

    2023

  4. [12]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  5. [13]

    Open-vocabulary object detection via vision and language knowledge distillation

    Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2023. 2

    2023

  6. [14]

    Grounded language-image pre-training

    Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages ...

    2022

  7. [15]

    Language-driven semantic seg- mentation

    Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic seg- mentation. In International Conference on Learning Rep- resentations, 2023. 2

    2023

  8. [16]

    Groupvit: Semantic segmentation emerges from text supervision

    Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 18134–18144, 2022. 2

    2022

  9. [17]

    Clip4clip: An empirical study of clip for end to end video clip retrieval

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. Neurocomputing, 508:293–304, 2021. 2

    2021

  10. [18]

    Videoclip: Con- trastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao (Bernie) Huang, Dmytro Okhonko, Armen Aghajanyan, and Florian Metze Luke Zettlemoyer Christoph Feichtenhofer. Videoclip: Con- trastive pre-training for zero-shot video-text understanding. In Conference on Empirical Methods in Natural Language Process...

    2021

  11. [19]

    Vqgan-clip: Open domain image generation and editing with natural language guidance

    Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Ed- ward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In European Con- ference on Computer Vision, 2022. 2

    2022

  12. [20]

    Clipdraw: Exploring text-to-drawing synthesis through language-image encoders

    Kevin Frans, Lisa Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. Advances in Neural Information Processing Sys- tems, 35:5207–5218, 2022

    2022

  13. [22]

    Clipasso: Semantically-aware object sketching

    Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Ro- man Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aware object sketching. ACM Transactions on Graphics (TOG), 41(4):1–11, 2022. 2

    2022

  14. [23]

    Region- aware pretraining for open-vocabulary object detection with vision transformers

    Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region- aware pretraining for open-vocabulary object detection with vision transformers. 2023 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 11144– 11154, 2023. 2

    2023

  15. [24]

    Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts

    Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts. In International Conference on Machine Learning , pages 25994–26009. PMLR, 2022. 2 9

    2022

  16. [25]

    Improving clip training with language rewrites

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Sys- tems, 36:35544–35575, 2023. 3

    2023

  17. [26]

    Stablerep: Synthetic images from text- to-image models make strong visual representation learn- ers

    Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text- to-image models make strong visual representation learn- ers. Advances in Neural Information Processing Systems , 36:48382–48402, 2023

    2023

  18. [27]

    Alip: Adaptive language-image pre-training with synthetic cap- tion

    Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziy- ong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic cap- tion. 2023 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 2910–2919, 2023

    2023

  19. [28]

    RLEG: Vision-language representation learn- ing with diffusion-based embedding generation

    Liming Zhao, Kecheng Zheng, Yun Zheng, Deli Zhao, and Jingren Zhou. RLEG: Vision-language representation learn- ing with diffusion-based embedding generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engel- hardt, Sivan Sabato, and Jonathan Scarlett, editors, ...

    2023

  20. [29]

    Veclip: Im- proving clip training via visual-enriched captions

    Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, and Meng Cao. Veclip: Im- proving clip training via visual-enriched captions. In Euro- pean Conference on Computer Vision, 2023. 3

    2023

  21. [30]

    Mllms- augmented visual-language representation learning

    Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You. Mllms- augmented visual-language representation learning. CoRR,

  22. [31]

    Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C

    Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014. 3

    2014

  23. [32]

    Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee

    Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. Generative ad- versarial text to image synthesis. InInternational Conference on Machine Learning, 2016. 3

    2016

  24. [33]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  25. [34]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125, 2022

    2022 arXiv

  26. [35]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  27. [36]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  28. [37]

    Sdedit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equa- tions. In International Conference on Learning Representa- tions, 2021. 3

    2021

  29. [38]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 3, 5

    2022 arXiv

  30. [39]

    Imagic: Text-based real image editing with diffusion mod- els

    Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Hui- Tang Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion mod- els. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6007–6017, 2022

    2023

  31. [40]

    Photoswap: Personalized subject swapping in images

    Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, Hyun- Joon Jung, et al. Photoswap: Personalized subject swapping in images. Advances in Neural Information Processing Sys- tems, 36:35202–35217, 2023. 3

    2023

  32. [41]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Interna- tional Conference on Machine Learning, 2021. 3

    2021

  33. [42]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18187–18197, 2021

    2022

  34. [43]

    Fleet, Radu Soricut, Jason Baldridge, Mo- hammad Norouzi, Peter Anderson, and William Chan

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Jason Baldridge, Mo- hammad Norouzi, Peter Anderson, and William Chan. Im- agen editor and editbench: Advancing and evaluati...

    2023

  35. [44]

    Text2live: Text-driven layered image and video editing

    Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vi- sion, pages 707–723. Springer, 2022

    2022

  36. [45]

    Diffedit: Diffusion-based semantic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. In ICLR 2023 (Eleventh Inter- national Conference on Learning Representations), 2023. 3

    2023

  37. [46]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1921–1930, 2022. 3, 6

    2023

  38. [47]

    Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and edit- ing

    Ming Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and edit- ing. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22503–22513, 2023. 3, 6

    2023

  39. [48]

    Goemo- tions: A dataset of fine-grained emotions

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemo- tions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547, 2020. 3 10

    2005 arXiv

  40. [49]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3

    2024

  41. [50]

    Long-clip: Unlocking the long-text capa- bility of clip

    Beichen Zhang, Pan Zhang, Xiao wen Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capa- bility of clip. In European Conference on Computer Vision,

  42. [51]

    Ot-clip: Un- derstanding and generalizing clip via optimal transport

    Liangliang Shi, Jack Fan, and Junchi Yan. Ot-clip: Un- derstanding and generalizing clip via optimal transport. In Forty-first International Conference on Machine Learning ,

  43. [52]

    Inversion-free image editing with natural language

    Sihan Xu, Yidong Huang, Jiayi Pan, Ziqiao Ma, and Joyce Chai. Inversion-free image editing with natural language. In Conference on Computer Vision and Pattern Recognition 2024, 2024. 5, 6

    2024

  44. [53]

    Senticap: Generating image descriptions with sentiments

    Alexander Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descriptions with sentiments. In Proceed- ings of the AAAI conference on artificial intelligence , vol- ume 30, 2016. 5

    2016

  45. [54]

    Guibas, and S

    Panos Achlioptas, Maks Ovsjanikov, Leonidas J. Guibas, and S. Tulyakov. Affection: Learning affective explana- tions for real-world visual data. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6641–6651, 2022. 5

    2023

  46. [55]

    Kulikov, and Tomer Michaeli

    Inbar Huberman-Spiegelglas, Vladimir B. Kulikov, and Tomer Michaeli. An edit friendly ddpm noise space: In- version and manipulations. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12469–12478, 2023. 6

    2024

  47. [56]

    Stylediffusion: Controllable disentangled style transfer via diffusion mod- els

    Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion mod- els. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7643–7655, 2023

    2023

  48. [57]

    Null-text inversion for editing real im- ages using guided diffusion models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real im- ages using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023

    2023

  49. [58]

    Pnp inversion: Boosting diffusion-based editing with 3 lines of code

    Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting diffusion-based editing with 3 lines of code. InThe Twelfth International Conference on Learning Representations, 2024. 6 11

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.