Pith. sign in

REVIEW 4 major objections 5 minor 91 references

FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FACEMUG generates realistic local facial edits from five input modalities at once, preserving unedited regions.

desk verdict A capable multimodal local-editing system with a real fairness problem in its headline comparisons and an under-supported warping module. read the letter →

arxiv 2412.19009 v1 pith:RKUM7GAW submitted 2024-12-26 cs.CV cs.MM

Wanglong Lu , Jikai Wang , Xiaogang Jin , Xianta Jiang , Hanli Zhao This is my paper · ORCID
classification cs.CVcs.MM
keywords facialimageeditingmultimodalsynthesisgenerativeadversarialnetworkslocallatentspacefusionself-supervisedwarpingStyleGANimage-to-imagetranslation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FACEMUG is a single GAN-based framework for local facial editing that accepts any combination of five modalities—sketches, semantic maps, color maps, exemplar images, and text or attribute labels—plus a mask that marks the region to edit. The paper claims it is the first method to generate realistic facial features on the edited regions while keeping visual coherence with the unedited background, and to do so without degrading after repeated incremental edits. If true, this would let users interactively edit specific facial attributes one step at a time, guided by whatever inputs they happen to have, without retraining per modality or collecting manually paired training data.

What carries the argument

The central object is the unified generative latent space $\mathcal{W}^+$ of StyleGAN V2, which receives all modalities through a shared style encoder. The multimodal aggregation module merges heterogeneous pixel-wise inputs into a homogeneous feature space using per-modality residual blocks and a softmax-normalized adaptive weighting scheme; the multimodal generator then fuses these features with StyleGAN facial priors using style fusion blocks that compute spatially-variant gate maps. The self-supervised latent warping network, built from code-to-code modulation blocks with channel and position cross-attention plus gating, learns pose transfer from augmented and flipped images under identity, LPIPS, and attribute losses; the diversity-enhanced attribute loss uses style mixing and interpolation to expose the model to a wide range of latent codes during training.

What would settle it

A controlled comparison would take a set of free-hand sketches and hand-edited semantic maps, condition FACEMUG on them versus on the automatically generated equivalents, and measure output quality with FID, LPIPS, or user preference; if quality drops substantially for the real user inputs, the claim that training requires no manual annotation would be weakened.

Watch

Extended reading notes

Core claim

The authors claim that aligning all modalities into the StyleGAN $\mathcal{W}^+$ latent space, then fusing them with a multimodal aggregation module and a generator that draws on a pre-trained StyleGAN as a facial feature bank, yields globally consistent local facial edits that respond to multimodal guidance. A self-supervised latent warping module transfers the pose of the edited image onto exemplar latent codes without pose detectors or manual annotations, and a diversity-enhanced attribute loss simulates latent editing during training. Extensive comparisons to diffusion-based and other multimodal editing methods show lower FID and LPIPS scores when masks are used, faster inference (about 34 FPS on a consumer GPU), and no quality collapse in incremental editing scenarios.

Load-bearing premise

The training modalities—sketches, semantic maps, and colors—are generated automatically, and editing quality on real user-drawn inputs depends on those auto-generated inputs being representative of what users actually provide at inference time.

Editorial extensions

If this is right

  • Users can edit a face in several successive steps, each changing only the masked region, without the quality collapse that the paper demonstrates for existing multimodal methods.
  • A single trained model covers all combinations of the five modalities; adding a modality does not require training a separate uni-modal network or collecting paired annotations.
  • Text and attribute labels steer attributes semantically while sketches, semantic maps, and colors control geometry and texture in the same editing pass.
  • Exemplar-guided local editing remains coherent even when the exemplar's pose differs from the edited photo, because the latent warping module aligns poses in $\mathcal{W}^+$ space.
  • Inference runs at roughly 34 frames per second, making interactive local facial editing practical on consumer hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same architecture could be lifted from faces to other structured categories, such as full bodies, cars, or indoor scenes, because the aggregation and fusion mechanism makes no face-specific assumption beyond the pre-trained generator used as the feature bank.
  • A natural stress test is contradictory modalities, for example a sketch of closed eyes paired with a text prompt saying 'open eyes'; a learned per-modality confidence weighting trained on synthetic conflicting inputs could extend the aggregation module and likely improve robustness.
  • Because the latent warping module learns pose transfer without pose annotations, it could be applied to other latent-code transformation tasks such as video face reenactment or cross-view editing, where alignment in latent space replaces explicit 3D reconstruction.
  • The one-month training time on a single V100 is a practical bottleneck for adoption; testing a lightweight generator or progressive training schedule would reveal whether the claimed editing quality can survive reduced capacity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FACEMUG, a GAN-based framework for local facial editing conditioned on up to five modalities (sketch, semantic map, color map, exemplar image, text/attribute label) plus a mask. The pipeline maps inputs into StyleGAN W+ space, warps exemplar latents to match the target pose via a self-supervised latent warping module, and uses a multimodal aggregation module and style fusion blocks to generate the edited region; Eq. (1) then pastes generated pixels only inside the mask. The authors report FID/U-IDS/LPIPS comparisons against TediGAN, PoE-GAN, ColDiffusion, Unite&Conquer, and several uni-modal baselines, ablation studies, a user study, and qualitative incremental editing examples.

Significance. If the results hold, FACEMUG would be a useful contribution: it offers fast (29 ms inference), single-model multimodal local editing with exact preservation of unmasked pixels, and it avoids manually paired cross-modality annotations. The architecture is coherent, and the mask-composition formulation in Eq. (1) gives a by-construction guarantee that unedited pixels remain unchanged. The paper also reports extensive experiments and ablations, including a user study, and the self-supervised warping objective is an interesting idea. However, several of the central empirical claims—modality-matched superiority, cross-identity exemplar warping, and incremental editing—are not fully supported by the experiments as reported.

major comments (4)
  1. [Tables I, II, IV, V] The quantitative comparisons are not modality-matched: the largest improvements come from adding the mask input (+Ma), which baselines do not receive. For example, Table II shows FID falling from 29.96 (Ours +Sk+Se) to 10.36 (Ours +Sk+Se+Ma) on CelebA-HQ, and Table IV from 36.58 (Ours +Sk) to 11.43 (Ours +Sk+Ma). Because the mask supplies the exact ground-truth pixels outside the edit region, this is a strong additional conditioning signal unavailable to the baselines. The paper should either compare baselines under the same mask-conditioned protocol, or present the non-mask results as the primary modality-matched comparison and clearly state that the +Ma rows address a different task. As written, the headline claim that FACEMUG outperforms SOTA multimodal editing methods is inflated.
  2. [II.F.2, Eq. (10), Algorithm 2, Table VII] The latent warping module is trained exclusively on same-identity triplets: source codes are interpolations between w_ini and w_f of the same image, and the losses in Eq. (10) all anchor to w_ini. There is no term that encourages preservation of the source identity when w_so comes from a different person than the target, which is exactly the inference setting in Section II.B and Fig. 2. The evaluation in Table VII reports only FID and LPIPS for exemplar-guided editing, and the CSIM result in Section H compares frontalized images to ground truth, not output-to-exemplar identity. I therefore do not see evidence that the claimed cross-identity pose transfer and identity preservation are achieved. Please add cross-identity training pairs or a direct identity-preservation metric (e.g., ArcFace cosine similarity between I_out and I_ex) with and without warping.
  3. [II.F.4, Figs. 25 and 26] The training modalities are auto-generated (pencil-sketch filter, face-parsing semantic maps, mean region colors), while the interactive examples in Figs. 25 and 26 use free-hand sketches and hand-edited semantic maps. No quantitative evaluation is given for such free-form inputs, so the claim that FACEMUG supports interactive or free-hand editing is only qualitatively supported. Please report quantitative metrics on hand-drawn/hand-edited inputs or otherwise characterize the sensitivity of the system to this train/inference distribution shift.
  4. [III.B, Figs. 1, 8, 23] The motivating claim that existing methods degrade after several iterations of incremental editing is not backed by any quantitative multi-step evaluation. The paper shows qualitative incremental sequences but does not measure how FID, LPIPS, or background consistency evolve over edit rounds, nor does it compare the incremental protocol with baselines under the same protocol. This is load-bearing for the incremental-editing part of the central contribution; please add a quantitative incremental-editing experiment.
minor comments (5)
  1. [II.C] The stated range lambda_clip in [0.1, 1.0] is inconsistent with the default lambda_clip = 0.05 used in the experiments.
  2. [Appendix C and D] There are typos in the headings: 'discrinimator' should be 'discriminator' and 'peseudo-codes' should be 'pseudo-codes'.
  3. [Fig. 14] The x-axis is labeled 'Masked ratio' but the tick labels are not legible; please provide a clearer axis with visible tick values.
  4. [III.A] The paper states that the same training and testing splits were used for all experiments, but ControlNet was fine-tuned on FFHQ; please clarify whether all pre-trained baselines were trained on the same FFHQ training split as FACEMUG.
  5. [Table III] The user study mixes results with and without masks; please report the with-mask and without-mask preference percentages separately, since they evaluate different settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FACEMUG is an empirical GAN framework validated on held-out test splits and external baselines; the cited self-losses are defined in-line and are not load-bearing.

full rationale

FACEMUG's central claims — multimodal local editing, global consistency, and latent warping — are empirical system properties. The training procedure (Section II.F) optimizes reconstruction/identity/perceptual/adversarial losses against ground-truth images, and the reported FID, U-IDS, LPIPS, and user-study numbers are computed on CelebA-HQ/FFHQ test images that were not used for training, using standard external metrics. The pretrained components (StyleGAN2, e4e, CLIP, ArcFace, VGG) are fixed external tools, not evidence imported from the authors' own prior work. No fitted parameter is renamed as a prediction: for example, the latent warping loss in Eq. (10) is a direct training objective, not a quantity that is later reported as a 'predicted' result. The only self-referential element is the citation of the authors' prior inpainting paper [24] for the identity/attribute losses, but those losses are fully defined in Eqs. (11)-(13) and are standard L2/cosine losses, so the citation is not load-bearing. The potential concern that the warping module is trained only on same-identity triples while deployed for cross-identity exemplar transfer is a generalization/evidence limitation, not a circularity: the learned residual is not defined in terms of the test-time output. Overall, the paper's claims rest on held-out evaluation rather than on a self-referential derivation.

Assumptions & free parameters 11 free parameters · 6 assumptions · 0 invented entities

The central claims rest on hand-chosen loss weights and domain assumptions about the StyleGAN latent space and automatic training-data generation. No new physical or conceptual entities are hypothesized; the proposed modules are trainable networks whose evidence is internal to the paper's experiments.

free parameters (11)
  • lambda_latent (warping loss weight) = 0.1
    Weight in Eq. (10) balancing identity, LPIPS, and attribute losses during latent warping training.
  • lambda_id (identity loss weight) = 0.1
    Weight on ArcFace identity loss in the main generator objective, Eq. (14)/(23).
  • lambda_attr (attribute loss weight) = 0.1
    Weight on the diversity-enhanced attribute loss constraining consistency between edited image latent codes and target codes.
  • lambda_lpips (perceptual loss weight) = 0.5
    Weight on LPIPS similarity between edited output and ground truth.
  • lambda_clip (CLIP directional loss weight) = 0.05, though text states range [0.1, 1.0]
    Balances directional CLIP loss in the text-driven latent optimization, Eq. (4). The stated default contradicts the stated allowed range.
  • lambda_reg (text optimization regularization weight) = 0.08
    Regularization term in Eq. (4) that keeps the optimized latent code near the starting code.
  • rho (exemplar source probability) = 0.5
    Threshold in Algorithm 3 controlling whether the exemplar is the ground-truth image or a random training image.
  • omega (modality keep probability) = 0.8
    Threshold controlling random dropout of individual modalities during training to improve robustness to missing inputs.
  • gamma (R1 regularization weight) = 10
    Weight for the R1 gradient penalty in the adversarial loss, Eq. (22).
  • text optimization schedule = 100-300 iterations, learning rate 0.1
    Gradient descent schedule for text-driven editing; no convergence guarantee or validation of edit quality across different prompts.
  • main training schedule = 800k iterations, batch size 8, learning rate 0.002
    Adversarial training schedule for aggregation module, refinement auto-encoder, and discriminator.
assumptions (6)
  • domain assumption The pretrained StyleGAN2 W+ latent space is sufficiently disentangled and expressive to act as a unified generative space for sketches, semantic maps, colors, exemplars, text, and attributes.
    The entire method maps all modalities into W+ and uses StyleGAN as a facial feature bank. If W+ cannot represent the conditional information, the fusion and editing pipeline fails. Invoked throughout Section II.
  • domain assumption Automatically generated sketches, semantic maps, and color maps are representative of user-provided free-hand and hand-edited inputs.
    Training uses a pencil-sketch filter, a face-parsing model, and mean region colors (Section II.F.4), while qualitative tests show free-hand sketches and hand-edited maps. Distribution shift is not measured.
  • domain assumption Pretrained StyleGAN2, e4e, ArcFace, VGG, and CLIP models provide reliable fixed feature extractors for face editing.
    These frozen models supply the latent space, identity metric, perceptual metric, and text-image alignment. Failures or biases in any of them propagate into editing results.
  • domain assumption The directional CLIP loss in Eq. (5) correctly captures semantically meaningful text-driven facial edits.
    Text editing relies on optimizing latent codes with the CLIP directional loss, without evidence that CLIP directions are disentangled for local facial attributes.
  • domain assumption Mirror-flipped and augmented images provide a self-supervised training signal that separates identity from pose in W+.
    The latent warping module is trained to preserve identity relative to wini while adopting the pose of wta. This assumes the required disentanglement exists and is learnable from the constructed triplets.
  • domain assumption Generating content only inside the mask, conditioned on unmasked pixels, yields globally consistent results without boundary artifacts.
    Eq. (1) guarantees unedited pixels remain unchanged, but visual coherence at the mask boundary depends on the generator's capacity and training distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing." pith.science (2026). https://pith.science/paper/RKUM7GAW

@misc{pith2026241219009,
  author       = {Pith},
  title        = {Pith review of: FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RKUM7GAW}},
  note         = {Machine review of arXiv:2412.19009}
}
read the original abstract

Existing facial editing methods have achieved remarkable results, yet they often fall short in supporting multimodal conditional local facial editing. One of the significant evidences is that their output image quality degrades dramatically after several iterations of incremental editing, as they do not support local editing. In this paper, we present a novel multimodal generative and fusion framework for globally-consistent local facial editing (FACEMUG) that can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. Different modalities, including sketches, semantic maps, color maps, exemplar images, text, and attribute labels, are adept at conveying diverse conditioning details, and their combined synergy can provide more explicit guidance for the editing process. We thus integrate all modalities into a unified generative latent space to enable multimodal local facial edits. Specifically, a novel multimodal feature fusion mechanism is proposed by utilizing multimodal aggregation and style fusion blocks to fuse facial priors and multimodalities in both latent and feature spaces. We further introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features, efficiently transferring the pose of the edited image to the given latent codes. We evaluate our FACEMUG through extensive experiments and comparisons to state-of-the-art (SOTA) methods. The results demonstrate the superiority of FACEMUG in terms of editing quality, flexibility, and semantic control, making it a promising solution for a wide range of local facial editing tasks.

Figures

Figures reproduced from arXiv: 2412.19009 by the authors.

Figure 1
Figure 1. Examples demonstrating the superior performance of FACEMUG in high-quality globally consistent local facial editing, using subsets of the five [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overall pipeline of our FACEMUG globally-consistent local facial editing: the given attribute label (or text), random latent code [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of our style fusion block. Conditioned by the modulated [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (21 more)
Figure 5
Figure 5. Figure 5: Visual comparison to ColDiffusion [8] and Unite&Conquer [9] for text-driven multimodal facial editing. Our method produces visually appealing and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visual comparison to PoE-GAN [7]. We used the input modalities and results published in their paper. The consistency between generated images [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Visual comparison to Unite&Conquer [9]. We show more FACEMUG results by adding extra masks. Our method shows better visual quality and [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison of incremental editing (the later editing taking the previous output image as input), compared to SOTA methods [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Visual comparison between our FACEMUG and the SOTA sketch-guided editing methods (pSp [33], DeepFaceEditing [34], and ControlNet [35]). [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Visual comparison between our FACEMUG and the SOTA semantic-guided editing methods (pSp [33], SEAN [38], SofGAN [37], SDM [40], and [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Visual comparison of the semantic-guided local facial editing with [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Visual performance of our FACEMUG with various modal inputs. The masked image was utilized for all outputs. There are a total of 32 combinations [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 13
Figure 13. Figure 13: Visual comparison of the ablation study on the latent warping module (from left to right in each group): the masked image, the exemplar image, [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Quantitative comparison of the ablation study on the latent warping module on the FFHQ dataset. “+Ex w/o warping”: the exemplar image without [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: The architecture of the latent warping network. [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 17
Figure 17. Figure 17: Visual comparison of color-guided facial editing between SC-FEGAN [22] and ours: (first two rows) sub-images represent the colors and masked [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Visual comparison of attribute-conditional editing between HFGI [65] and our FACEMUG. For FACEMUG, the left sub-images in each group [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: Visual comparison of our FACEMUG to the SOTA exemplar [PITH_FULL_IMAGE:figures/full_fig_p017_19.png]
Figure 20
Figure 20. Figure 20: Visual comparison to StyleFlow [53] on pose editing. Our warping module shows better visual quality and visual consistency for facial attributes. [PITH_FULL_IMAGE:figures/full_fig_p018_20.png]
Figure 21
Figure 21. Figure 21: Visual comparison to pSp [33] on face frontalization. Our warping module provides superior visual quality and maintains the consistency of attributes. [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 22
Figure 22. Figure 22: Visual comparison to FaceReenactment [71] on pose transfer. Our warping module ensures enhanced visual quality and consistency of facial features. [PITH_FULL_IMAGE:figures/full_fig_p018_22.png]
Figure 23
Figure 23. Figure 23: Incremental local facial editing examples with our FACEMUG. Each row: given an input image (first row), FACEMUG incrementally edits the facial [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]
Figure 24
Figure 24. Figure 24: More multimodal local facial editing examples with our FACEMUG. Sketches, semantic maps, colors, exemplars, and attribute labels were utilized [PITH_FULL_IMAGE:figures/full_fig_p019_24.png]
Figure 25
Figure 25. Figure 25: Examples of sketch-guided facial attribute editing with FACEMUG: texture editing, structure editing, hairstyle editing, and expression editing. For [PITH_FULL_IMAGE:figures/full_fig_p019_25.png]
Figure 26
Figure 26. Figure 26: Examples of semantic-guided facial attribute editing with FACEMUG: chin editing, hairstyle editing, hair removal, and accessory addition. For each [PITH_FULL_IMAGE:figures/full_fig_p019_26.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 69 canonical work pages

  1. [1]

    Identity-aware and shape- aware propagation of face editing in videos,

    Y .-R. Jiang, S.-Y . Chen, H. Fu, and L. Gao, “Identity-aware and shape- aware propagation of face editing in videos,” TVCG, pp. 1–12, 2023

    2023

  2. [2]

    Reenactartface: Artistic face image reenactment,

    L. Qu, J. Shang, X. Han, and H. Fu, “Reenactartface: Artistic face image reenactment,” TVCG, pp. 1–13, 2023

    2023

  3. [3]

    Caricatureshop: Personalized and photorealistic caricature sketching,

    X. Han, K. Hou, D. Du, Y . Qiu, S. Cui, K. Zhou, and Y . Yu, “Caricatureshop: Personalized and photorealistic caricature sketching,” TVCG, vol. 26, no. 7, pp. 2349–2361, 2020

    2020

  4. [4]

    Cross-domain and disentangled face manipulation with 3d guidance,

    C. Wang, M. Chai, M. He, D. Chen, and J. Liao, “Cross-domain and disentangled face manipulation with 3d guidance,” TVCG, vol. 29, no. 4, p. 2053–2066, 2023

    2023

  5. [5]

    Drawinginstyles: Por- trait image generation and editing with spatially conditioned stylegan,

    W. Su, H. Ye, S.-Y . Chen, L. Gao, and H. Fu, “Drawinginstyles: Por- trait image generation and editing with spatially conditioned stylegan,” TVCG, vol. 29, no. 10, pp. 4074–4088, 2023

    2023

  6. [6]

    Tedigan: Text-guided diverse face image generation and manipulation,

    W. Xia, Y . Yang, J.-H. Xue, and B. Wu, “Tedigan: Text-guided diverse face image generation and manipulation,” in CVPR, 2021, pp. 2256– 2265

    2021

  7. [7]

    Multimodal condi- tional image synthesis with product-of-experts gans,

    X. Huang, A. Mallya, T.-C. Wang, and M.-Y . Liu, “Multimodal condi- tional image synthesis with product-of-experts gans,” in ECCV, 2022, p. 91–109

    2022

  8. [8]

    Collaborative diffusion for multi-modal face generation and editing,

    Z. Huang, K. C. Chan, Y . Jiang, and Z. Liu, “Collaborative diffusion for multi-modal face generation and editing,” in CVPR, 2023, pp. 6080– 6090. 19 + Curl hair + Age Ground-truth Three modalities Out put Ground-truth Four modalities Ou tput Ground-truth Five modalities Output Fig. 24. More multimodal local facial editing examples with our FACEMUG. Sketc...

    2023

Show all 91 references
  1. [9]

    Unite and conquer: Plug & play multi-modal synthesis using diffusion models,

    N. G. Nair, W. G. C. Bandara, and V . M. Patel, “Unite and conquer: Plug & play multi-modal synthesis using diffusion models,” in CVPR, 2023, pp. 6070–6079

    2023

  2. [10]

    Generative adversarial nets,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial nets,” in NeurIPS, vol. 27, 2014, pp. 2672–2680

    2014

  3. [11]

    Analyzing and improving the image quality of stylegan,

    T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in CVPR, 2020, pp. 8107–8116

    2020

  4. [12]

    Glean: Generative latent bank for large-factor image super-resolution,

    K. C. Chan, X. Wang, X. Xu, J. Gu, and C. C. Loy, “Glean: Generative latent bank for large-factor image super-resolution,” in CVPR, 2021, pp. 14 245–14 254

    2021

  5. [13]

    Fader networks:manipulating images by sliding attributes,

    G. Lample, N. Zeghidour, N. Usunier, A. Bordes, L. DENOYER, and M. A. Ranzato, “Fader networks:manipulating images by sliding attributes,” in NeurIPS, vol. 30, 2017

    2017

  6. [14]

    A style-based generator architecture for generative adversarial networks,

    T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” TPAMI, vol. 43, no. 12, pp. 4217– 4228, 2021

    2021

  7. [15]

    Alias-free generative adversarial networks,

    T. Karras, M. Aittala, S. Laine, E. H ¨ark¨onen, J. Hellsten, J. Lehtinen, and T. Aila, “Alias-free generative adversarial networks,” in NeurIPS, vol. 34, 2021, pp. 852–863

    2021

  8. [16]

    Deep unsupervised learning using nonequilibrium thermodynamics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML, vol. 37, 2015, pp. 2256–2265

    2015

  9. [17]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, vol. 33, 2020, pp. 6840–6851

    2020

  10. [18]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” in NeurIPS, vol. 34, 2021, pp. 8780–8794

    2021

  11. [19]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10 684–10 695

    2022

  12. [20]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in ICLR, 2021

    2021

  13. [21]

    Score-based generative modeling in latent space,

    A. Vahdat, K. Kreis, and J. Kautz, “Score-based generative modeling in latent space,” in NeurIPS, vol. 34, 2021, pp. 11 287–11 302

    2021

  14. [22]

    Sc-fegan: Face editing generative adversarial network with user’s sketch and color,

    Y . Jo and J. Park, “Sc-fegan: Face editing generative adversarial network with user’s sketch and color,” in ICCV, 2019, pp. 1745–1753

    2019

  15. [23]

    Coarse-to-fine: facial structure editing of portrait images via latent space classifications,

    Y . Wu, Y . Yang, Q. Xiao, and X. Jin, “Coarse-to-fine: facial structure editing of portrait images via latent space classifications,” TOG, vol. 40, no. 4, p. Article 46, 2021

    2021

  16. [24]

    Do inpainting yourself: Generative facial inpainting guided by 20 exemplars,

    W. Lu, H. Zhao, X. Jiang, X. Jin, Y . Yang, M. Wang, J. Lyu, and K. Shi, “Do inpainting yourself: Generative facial inpainting guided by 20 exemplars,” arXiv preprint, vol. arXiv:2202.06358, 2022

    2022

  17. [25]

    Contextual residual aggregation for ultra high-resolution image inpainting,

    Z. Yi, Q. Tang, S. Azizi, D. Jang, and Z. Xu, “Contextual residual aggregation for ultra high-resolution image inpainting,” in CVPR, 2020, pp. 7508–7517

    2020

  18. [26]

    Grig: Few-shot generative residual image inpainting,

    W. Lu, X. Jiang, X. Jin, Y .-L. Yang, M. Gong, T. Wang, K. Shi, and H. Zhao, “Grig: Few-shot generative residual image inpainting,” arXiv preprint, vol. arXiv:2304.12035, 2023

    2023 arXiv

  19. [27]

    Large scale image completion via co-modulated generative adversarial networks,

    S. Zhao, J. Cui, Y . Sheng, Y . Dong, X. Liang, E. I. Chang, and Y . Xu, “Large scale image completion via co-modulated generative adversarial networks,” in ICLR, 2021

    2021

  20. [28]

    Faceshop: Deep sketch-based face image editing,

    T. Portenier, Q. Hu, A. Szab ´o, S. A. Bigdeli, P. Favaro, and M. Zwicker, “Faceshop: Deep sketch-based face image editing,” TOG, vol. 37, no. 4, 2018

    2018

  21. [29]

    Foreground-aware image inpainting,

    W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo, “Foreground-aware image inpainting,” in CVPR, 2019, pp. 5833–5841

    2019

  22. [30]

    Structureflow: Image inpainting via structure-aware appearance flow,

    Y . Ren, X. Yu, R. Zhang, T. H. Li, S. Liu, and G. Li, “Structureflow: Image inpainting via structure-aware appearance flow,” in ICCV, 2019, pp. 181–190

    2019

  23. [31]

    Stargan: Unified generative adversarial networks for multi-domain image-to- image translation,

    Y . Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to- image translation,” in CVPR, 2018, pp. 8789–8797

    2018

  24. [32]

    Stargan v2: Diverse image synthesis for multiple domains,

    Y . Choi, Y . Uh, J. Yoo, and J.-W. Ha, “Stargan v2: Diverse image synthesis for multiple domains,” in CVPR, 2020, pp. 8185–8194

    2020

  25. [33]

    Encoding in style: a stylegan encoder for image-to- image translation,

    E. Richardson, Y . Alaluf, O. Patashnik, Y . Nitzan, Y . Azar, S. Shapiro, and D. Cohen-Or, “Encoding in style: a stylegan encoder for image-to- image translation,” in CVPR, 2021, pp. 2287–2296

    2021

  26. [34]

    DeepFaceEditing: Deep face generation and editing with disentangled geometry and appearance control,

    S.-Y . Chen, F.-L. Liu, Y .-K. Lai, P. L. Rosin, C. Li, H. Fu, and L. Gao, “DeepFaceEditing: Deep face generation and editing with disentangled geometry and appearance control,” TOG, vol. 40, no. 4, pp. 90:1–90:15, 2021

    2021

  27. [35]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in ICCV, 2023, pp. 3836–3847

    2023

  28. [36]

    Maskgan: Towards diverse and interactive facial image manipulation,

    C.-H. Lee, Z. Liu, L. Wu, and P. Luo, “Maskgan: Towards diverse and interactive facial image manipulation,” in CVPR, 2020, pp. 5548–5557

    2020

  29. [37]

    Sofgan: A portrait image generator with dynamic styling,

    A. Chen, R. Liu, L. Xie, Z. Chen, H. Su, and J. Yu, “Sofgan: A portrait image generator with dynamic styling,” TOG, vol. 41, no. 1, p. Article 1, 2022

    2022

  30. [38]

    Sean: Image synthesis with semantic region-adaptive normalization,

    P. Zhu, R. Abdal, Y . Qin, and P. Wonka, “Sean: Image synthesis with semantic region-adaptive normalization,” in CVPR, 2020

    2020

  31. [39]

    Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing,

    Y . Shi, X. Yang, Y . Wan, and X. Shen, “Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing,” in CVPR, 2022, pp. 11 254–11 264

    2022

  32. [40]

    Semantic image synthesis via diffusion models,

    W. Wang, J. Bao, W. Zhou, D. Chen, D. Chen, L. Yuan, and H. Li, “Semantic image synthesis via diffusion models,” arXiv preprint arXiv:2207.00050, 2022

    2022

  33. [41]

    Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis,

    J. Sun, X. Wang, Y . Shi, L. Wang, J. Wang, and Y . Liu, “Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis,” TOG, vol. 41, no. 6, 2022

    2022

  34. [42]

    Elegant: Exchanging latent encodings with gan for transferring multiple face attributes,

    T. Xiao, J. Hong, and J. Ma, “Elegant: Exchanging latent encodings with gan for transferring multiple face attributes,” in ECCV, 2018, pp. 172–187

    2018

  35. [43]

    Mulgan: Facial attribute editing by exemplar,

    J. Guo, Z. Qian, Z. Zhou, and Y . Liu, “Mulgan: Facial attribute editing by exemplar,” arXiv preprint, vol. arXiv:1912.12396, 2019

    1912 arXiv

  36. [44]

    Advancing high fidelity identity swapping for forgery detection,

    L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, “Advancing high fidelity identity swapping for forgery detection,” inCVPR, 2020, pp. 5073–5082

    2020

  37. [45]

    Learning to warp for style transfer,

    X. Liu, Y . Yang, and P. Hall, “Learning to warp for style transfer,” in CVPR, 2021, pp. 3701–3710

    2021

  38. [46]

    Simswap: An efficient framework for high fidelity face swapping,

    R. Chen, X. Chen, B. Ni, and Y . Ge, “Simswap: An efficient framework for high fidelity face swapping,” in ACM MM, 2020, pp. 2003–2011

    2020

  39. [47]

    Image-to-image translation via hierarchical style disentan- glement,

    X. Li, S. Zhang, J. Hu, L. Cao, X. Hong, X. Mao, F. Huang, Y . Wu, and R. Ji, “Image-to-image translation via hierarchical style disentan- glement,” in CVPR, 2021, pp. 8639–8648

    2021

  40. [48]

    Guided image inpainting: Replacing an image region by pulling content from another image,

    Y . Zhao, B. Price, S. Cohen, and D. Gurari, “Guided image inpainting: Replacing an image region by pulling content from another image,” in WACV, 2019, pp. 1514–1523

    2019

  41. [49]

    Image re-composition via regional content-style decoupling,

    R. Zhang, W. Li, Y . Zhang, H. Zhang, J. Yu, R. Yang, and W. Xu, “Image re-composition via regional content-style decoupling,” in ACM MM, 2021, pp. 3–11

    2021

  42. [50]

    Exploiting spatial dimensions of latent in gan for real-time image editing,

    H. Kim, Y . Choi, J. Kim, S. Yoo, and Y . Uh, “Exploiting spatial dimensions of latent in gan for real-time image editing,” in CVPR, 2021, pp. 852–861

    2021

  43. [51]

    Hairmapper: Removing hair from portraits using gans,

    Y . Wu, Y . Yang, and X. Jin, “Hairmapper: Removing hair from portraits using gans,” in CVPR, 2022, pp. 4227–4236

    2022

  44. [52]

    Interfacegan: Interpreting the disentangled face representation learned by gans,

    Y . Shen, C. Yang, X. Tang, and B. Zhou, “Interfacegan: Interpreting the disentangled face representation learned by gans,” TPAMI, vol. 44, no. 4, pp. 2004–2018, 2022

    2004

  45. [53]

    Styleflow: Attribute- conditioned exploration of stylegan-generated images using conditional continuous normalizing flows,

    R. Abdal, P. Zhu, N. J. Mitra, and P. Wonka, “Styleflow: Attribute- conditioned exploration of stylegan-generated images using conditional continuous normalizing flows,” TOG, vol. 40, no. 3, may 2021

    2021

  46. [54]

    Clip2stylegan: Unsupervised extraction of stylegan edit directions,

    R. Abdal, P. Zhu, J. Femiani, N. Mitra, and P. Wonka, “Clip2stylegan: Unsupervised extraction of stylegan edit directions,” in SIGGRAPH, 2022, pp. 48:1–48:9

    2022

  47. [55]

    Closed-form factorization of latent semantics in gans,

    Y . Shen and B. Zhou, “Closed-form factorization of latent semantics in gans,” in CVPR, 2020, pp. 1532–1540

    2020

  48. [56]

    Ganspace: Discovering interpretable gan controls,

    E. H ¨ark¨onen, A. Hertzmann, J. Lehtinen, and S. Paris, “Ganspace: Discovering interpretable gan controls,” in NeurIPS, vol. 33, 2020, pp. 9841–9850

    2020

  49. [57]

    Styleclip: Text-driven manipulation of stylegan imagery,

    O. Patashnik, Z. Wu, E. Shechtman, D. Cohen-Or, and D. Lischinski, “Styleclip: Text-driven manipulation of stylegan imagery,” in ICCV, 2021, pp. 2085–2094

    2021

  50. [58]

    One model to edit them all: Free-form text-driven image manipulation with semantic modulations,

    Y . Zhu, H. Liu, Y . Song, Z. Yuan, X. Han, C. Yuan, Q. Chen, and J. Wang, “One model to edit them all: Free-form text-driven image manipulation with semantic modulations,” in NeurIPS, vol. 35, 2022, pp. 25 146–25 159

    2022

  51. [59]

    Clip-guided stylegan inversion for text-driven real image editing,

    A. C. Baykal, A. B. Anees, D. Ceylan, E. Erdem, A. Erdem, and D. Yuret, “Clip-guided stylegan inversion for text-driven real image editing,” TOG, vol. 42, no. 5, 2023

    2023

  52. [60]

    Image2stylegan: How to embed images into the stylegan latent space?

    R. Abdal, Y . Qin, and P. Wonka, “Image2stylegan: How to embed images into the stylegan latent space?” in ICCV, 2019, pp. 4431–4440

    2019

  53. [61]

    Image2stylegan++: How to edit the embedded images?

    ——, “Image2stylegan++: How to edit the embedded images?” in CVPR, 2020, pp. 8293–8302

    2020

  54. [62]

    Pivotal tuning for latent-based editing of real images,

    D. Roich, R. Mokady, A. H. Bermano, and D. Cohen-Or, “Pivotal tuning for latent-based editing of real images,” TOG, vol. 42, no. 1, 2022

    2022

  55. [63]

    In-domain gan inversion for real image editing,

    J. Zhu, Y . Shen, D. Zhao, and B. Zhou, “In-domain gan inversion for real image editing,” in ECCV, 2020, pp. 592–608

    2020

  56. [64]

    Designing an encoder for stylegan image manipulation,

    O. Tov, Y . Alaluf, Y . Nitzan, O. Patashnik, and D. Cohen-Or, “Designing an encoder for stylegan image manipulation,” TOG, vol. 40, no. 4, p. Article 133, 2021

    2021

  57. [65]

    High-fidelity gan inversion for image attribute editing,

    T. Wang, Y . Zhang, Y . Fan, J. Wang, and Q. Chen, “High-fidelity gan inversion for image attribute editing,” inCVPR, 2022, pp. 11 379–11 388

    2022

  58. [66]

    Reganie: Rectifying gan inversion errors for accurate real image editing,

    B. Li, T. Ma, P. Zhang, M. Hua, W. Liu, Q. He, and Z. Yi, “Reganie: Rectifying gan inversion errors for accurate real image editing,” AAAI, vol. 37, no. 1, pp. 1269–1277, 2023

    2023

  59. [67]

    Variational mixture- of-experts autoencoders for multi-modal deep generative models,

    Y . Shi, N. Siddharth, B. Paige, and P. H. S. Torr, “Variational mixture- of-experts autoencoders for multi-modal deep generative models,” in NeurIPS, 2019

    2019

  60. [68]

    Generalized multimodal ELBO,

    T. M. Sutter, I. Daunhawer, and J. E. V ogt, “Generalized multimodal ELBO,” in ICLR, 2021

    2021

  61. [69]

    Joint multimodal learning with deep generative models,

    M. Suzuki, K. Nakayama, and Y . Matsuo, “Joint multimodal learning with deep generative models,” in ICLRW, 2017

    2017

  62. [70]

    Multimodal image synthesis and editing: The generative ai era,

    F. Zhan, Y . Yu, R. Wu, J. Zhang, S. Lu, L. Liu, A. Kortylewski, C. Theobalt, and E. Xing, “Multimodal image synthesis and editing: The generative ai era,” TPAMI, vol. 45, no. 12, pp. 15 098–15 119, 2023

    2023

  63. [71]

    Finding directions in gan’s latent space for neural face reenactment,

    S. Bounareli, V . Argyriou, and G. Tzimiropoulos, “Finding directions in gan’s latent space for neural face reenactment,” BMVC, 2022

    2022

  64. [72]

    Stylerig: Rigging stylegan for 3d control over portrait images,

    A. K. Tewari, M. A. Elgharib, G. Bharaj, F. Bernard, H.-P. Seidel, P. P ´erez, M. Zollh ¨ofer, and C. Theobalt, “Stylerig: Rigging stylegan for 3d control over portrait images,” CVPR, pp. 6141–6150, 2020

    2020

  65. [73]

    Interpreting the latent space of gans for semantic face editing,

    Y . Shen, J. Gu, X. Tang, and B. Zhou, “Interpreting the latent space of gans for semantic face editing,” in CVPR, 2020

    2020

  66. [74]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763

    2021

  67. [75]

    Clip2protect: Protecting facial privacy using text-guided makeup via adversarial latent search,

    F. Shamshad, M. Naseer, and K. Nandakumar, “Clip2protect: Protecting facial privacy using text-guided makeup via adversarial latent search,” in CVPR, 2023, pp. 20 595–20 605

    2023

  68. [76]

    Spatially multi-conditional image generation,

    N. Popovi ´c, R. Chakraborty, D. P. Paudel, T. Probst, and L. Van Gool, “Spatially multi-conditional image generation,” in WACV, 2023, pp. 734–743

    2023

  69. [77]

    Differentiable augmen- tation for data-efficient gan training,

    S. Zhao, Z. Liu, J. Lin, J.-Y . Zhu, and S. Han, “Differentiable augmen- tation for data-efficient gan training,” in NeurIPS, vol. 33, 2020, pp. 7559–7570

    2020

  70. [78]

    Arcface: Additive angular margin loss for deep face recognition,

    J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in CVPR, 2019, pp. 4685–4694

    2019

  71. [79]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018, pp. 586–595

    2018

  72. [80]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in ICLR, 2015

    2015

  73. [81]

    Feat: Face editing with attention,

    X. Hou, L. Shen, O. Patashnik, D. Cohen-Or, and H. Huang, “Feat: Face editing with attention,” arXiv preprint, vol. arXiv:2202.02713, 2022. 21

    2022 arXiv

  74. [82]

    Which training methods for gans do actually converge?

    L. Mescheder, S. Nowozin, and A. Geiger, “Which training methods for gans do actually converge?” in ICML, vol. 80, 2018, pp. 3481–3490

    2018

  75. [83]

    Face parsing in pytorch,

    zllrunning, “Face parsing in pytorch,” 2023, [Online; accessed 14-Feb- 2023]. [Online]. Available: https://github.com/zllrunning/face-parsing. PyTorch

    2023

  76. [84]

    Progressive growing of GANs for improved quality, stability, and variation,

    T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of GANs for improved quality, stability, and variation,” in ICLR, 2018

    2018

  77. [85]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in NeurIPS, vol. 30, 2017, pp. 6629–6640

    2017

  78. [86]

    Image inpainting for irregular holes using partial convolutions,

    G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in ECCV, 2018, pp. 89–105

    2018

  79. [87]

    Talk-to-edit: Fine- grained facial editing via dialog,

    Y . Jiang, Z. Huang, X. Pan, C. C. Loy, and Z. Liu, “Talk-to-edit: Fine- grained facial editing via dialog,” in ICCV, 2021, pp. 13 799–13 808

    2021

  80. [88]

    Free-form image inpainting with gated convolution,

    J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. Huang, “Free-form image inpainting with gated convolution,” in ICCV, 2019, pp. 4470–4479

    2019

  81. [89]

    Vectortalker: Svg talking face generation with progressive vectorisation,

    H. Hu, X. Wang, J. Sun, Y . Fan, Y . Guo, and C. Jiang, “Vectortalker: Svg talking face generation with progressive vectorisation,” arXiv preprint, vol. arXiv:2312.11568, 2023

    2023 arXiv

  82. [90]

    Dual attention network for scene segmentation,

    J. Fu, J. Liu, H. Tian, Y . Li, Y . Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in CVPR, 2019, pp. 3146–3154

    2019

  83. [91]

    Ilvr: Conditioning method for denoising diffusion probabilistic models,

    J. Choi, S. Kim, Y . Jeong, Y . Gwon, and S. Yoon, “Ilvr: Conditioning method for denoising diffusion probabilistic models,” in ICCV, 2021, pp. 14 347–14 356

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.