Pith. sign in

REVIEW 4 major objections 5 minor 78 references

CFSynthesis: Controllable and Free-view 3D Human Video Synthesis

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CFSynthesis claims that one reference image, a texture-mapped 3D body model, and separated foreground/background controls can synthesize high-quality human videos from arbitrary viewpoints and user-chosen scene backgrounds.

desk verdict Plausible architecture, but the masking ablation is Disco's numbers in disguise and the SOTA claim rests on unevaluated comparisons. read the letter →

arxiv 2412.11067 v3 pith:OBJGD2ZN submitted 2024-12-15 cs.CV

Liyuan Cui , Xiaogang Xu , Wenqi Dong , Zesong Yang , Hujun Bao , Zhaopeng Cui This is my paper · ORCID
classification cs.CV
keywords humanvideosynthesistexturedSMPLrepresentationfree-viewpointgenerationlatentdiffusionposecontrolforeground-backgroundseparationsingle-imageanimationuser-controlledbackground
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that one photo can drive a person through complex 3D motions viewed from arbitrary camera angles, with the background replaced on request. Its two bets are that a textured 3D body model, rather than a skeleton or depth map, carries enough appearance information to keep the character recognizable as the camera moves, and that splitting the frame into foreground and background during generation prevents the flicker and identity drift that plague 2D animation. The authors report best-in-class numbers on the TikTok and AIST benchmarks and demonstrate free-viewpoint consistency against an existing 4D generator. A reader would care because the method promises a practical use case: any single portrait becomes an animatable, re-sceneable 3D character without multi-view capture.

What carries the argument

The load-bearing object is the textured SMPL pose representation: $M^i = \Omega(U_{\mathrm{com}}, \theta^i \cdot T^i)$, the projection of a pseudo-complete UV texture map onto an SMPL body at pose $\theta^i$ through camera trajectory $T^i$; SMPL is a standard skinned 3D human body model. The texture map is built by mapping reference-image pixels to surface coordinates and completing the unseen regions with inpainting, so the projected image already contains plausible color for every visible body part. Around it, the framework adds a pose extractor (convolutions plus an attention layer) that converts $M^i$ into a pose latent $z_{\mathrm{pose}}$; a foreground encoder that injects masked reference-image features $z^f_l = z_l \otimes f^{\mathrm{seg}}_l$ at each resolution; and a background encoder that turns a user scene sequence into $z_{\mathrm{bg}}$. The fusion step $Z_{\mathrm{full}} = \lambda\,\mathrm{Softmax}(QK_{\mathrm{bg}}^T/\sqrt{d})V_{\mathrm{fg}} + \mathrm{Softmax}(QK_{\mathrm{noise}}^T/\sqrt{d})V_{\mathrm{fg}}$ recombines identity and scene conditions during denoising, with only the pose extractor, foreground spatial attention, cross-attention, and background encoder trained while the rest of the diffusion network stays frozen.

What would settle it

Take a person whose reference photo hides part of the outfit, such as the back, sides, or occluded folds, generate views across 360 degrees, and compare against ground-truth multi-view footage of the same person; if the clothing pattern or face in the synthesized back views diverges measurably from the recording, the single-image texture prior is not supplying the claimed appearance information.

Watch

Extended reading notes

Core claim

CFSynthesis claims that a pseudo-complete UV texture map of a person, estimated from one reference image and completed by inpainting, can be warped onto an SMPL body sequence and projected through a user-defined camera trajectory to produce a pose signal that is both geometrically exact and appearance-rich. This signal is encoded by a lightweight pose extractor and concatenated with the noisy latent; a separate foreground encoder attaches reference appearance through spatial attention at multiple resolutions, while a background encoder embeds the user-selected scene. A masking mechanism multiplies the foreground latents by a downsampled binary mask at each resolution so features do not bleed over contour edges into the background region. The three streams are recomposed in the denoising U-Net through a cross-attention fusion step. With this design the paper claims the best L1, SSIM, LPIPS, FID-VID, and FVD on TikTok and the best SSIM, LPIPS, and FID on AIST, and it argues that the textured-SMPL prior is what makes novel viewpoints stable.

Load-bearing premise

The whole free-view guarantee rests on the assumption that the estimated 3D surface texture built from one photo is accurate enough to show what the person looks like from the back and sides; if that texture is incomplete or misaligned, the wrong colors and details are projected onto the body and identity consistency breaks.

Editorial extensions

If this is right

  • A single reference image is enough to synthesize the same character from novel viewpoints and under new camera trajectories, without multi-view training data.
  • Swapping the pose signal from 2D skeletons to textured SMPL reduces appearance error and stabilizes identity, which is what the paper's ablation on the SMPL representation claims.
  • Encoding foreground and background separately, with the masking mechanism, removes edge flicker and allows dynamic user-specified backgrounds rather than fixed ones.
  • Because only a few lightweight modules are trained on a small dataset, the method can be retargeted to in-the-wild 4D motions and language-generated motions with minimal cost.
  • If the reported metrics hold, CFSynthesis outperforms current 2D-animation baselines on TikTok and AIST across most image- and video-level fidelity measures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the practical quality ceiling is set by the single-image texture inpainting step: whenever the inpainter cannot guess the back of the body or occluded clothing, that error will appear directly as identity drift in the novel views.
  • A natural extension would be to plug in a stronger 3D texture estimator, one trained on multi-view data, and check whether the free-view metrics improve without touching the diffusion backbone.
  • The masking mechanism is generalizable: any latent-diffusion system that composites two content streams can suppress boundary bleeding by keeping each stream's features inside its own segmentation mask at every resolution.
  • Because the background encoder accepts any rendered sequence, the same framework could insert an animated person into a moving virtual camera in a 3D scene, not just into video backgrounds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents CFSynthesis, a stable-diffusion-based framework for human video synthesis that takes a single reference image, a 3D SMPL motion sequence, a camera trajectory, and an optional user-specified background as inputs. The two main technical components are (i) a textured SMPL representation obtained by warping a pseudo-complete UV texture map onto the SMPL body and projecting it under the target camera, injected through a pose extractor; and (ii) a foreground-background separation learning strategy with a masking mechanism that suppresses feature leakage across the human silhouette, together with a background encoder and a composed-decoding cross-attention fusion. The system is trained on TikTok and AIST and evaluated against recent animation methods, with additional qualitative free-view results and ablations. The paper claims state-of-the-art performance in complex human animation as well as effective free-view and background-controlled synthesis.

Significance. The design is timely and potentially useful: it extends 2D diffusion-based human animation toward free-viewpoint control while avoiding multi-view training data, and it integrates user-specified backgrounds in a single pipeline. The textured-SMPL pose representation is a sensible way to inject 3D priors, and the qualitative demonstrations of novel views, background insertion, and in-the-wild 4D synthesis are appealing. However, the quantitative evidence for the central SOTA claim is not yet convincing: the key ablation for the masking mechanism is invalid as reported, baseline numbers are copied from other papers without error bars, the free-view comparison with Human4Dit is qualitative and sourced from the competitor's website, and the paper itself acknowledges that the single-reference texture map can cause cross-perspective instability. With stronger evaluation the contribution could be solid; in its current form the evidence is insufficient to support the advertised claims.

major comments (4)
  1. [Sec. IV-C, Table V] The ablation for the masking mechanism is not valid as reported: the 'w/o' row reproduces exactly the DisCo row of Table II (PSNR 29.03, SSIM 0.668, LPIPS 0.292), while the 'w/' row is the full CFSynthesis model. This design does not compare the same architecture with and without the masking mechanism, so the table provides no evidence that masking is responsible for the improvement. Please re-run the ablation on the full model with the mask removed under identical training conditions.
  2. [Sec. IV-B, Tables II and III] The state-of-the-art claim rests on point estimates without error bars or significance tests, and several baseline numbers are copied from prior papers rather than measured under the protocol used for CFSynthesis. Because the margins on several metrics are small (e.g., SSIM 0.820 vs 0.811 for UniAnimate; LPIPS 0.200 vs 0.230), cross-protocol differences could be as large as the reported gaps. Please provide confidence intervals and re-run baselines under the same evaluation protocol, or state explicitly which numbers were not re-run.
  3. [Sec. IV-B, Fig. 7] The comparison with Human4Dit is qualitative, uses screenshots obtained from the competitor's website, and is not accompanied by any quantitative or controlled measurement. Since Human4Dit is non-open-source and the data are in-the-wild, the free-view superiority claim is not supported by this evidence. Please provide a quantitative evaluation on a common benchmark or a controlled test set with matched conditions.
  4. [Sec. III-B and Limitations] The central free-view mechanism relies on the pseudo-complete UV texture map U_com estimated from a single reference image via SMPLitex and frozen-Stable-Diffusion inpainting. The manuscript itself acknowledges that this single-reference estimation 'lead[s] to possible instabilities in generation quality across different perspectives.' Because the free-view claim depends on U_com, the paper should quantify its robustness, for example by measuring texture coverage or identity/consistency metrics on views that expose unseen body regions.
minor comments (5)
  1. [Fig. 1 and throughout] There are several spelling errors in figure captions and text, including 'backgorund' in Fig. 1, 'MagicAnimayte' in the Fig. 4 caption, and 'FIV-VID' in Sec. IV-C.
  2. [References [35] and [33]/[74]] Reference [35] is cited for T2I-Adapter but the entry lists IP-Adapter, and references [33] and [74] are duplicates of the same Stable Diffusion paper; please correct the citations.
  3. [Table IV and Fig. 3] The notation for the pose estimator is inconsistent: 'dwpose' and 'DWPose' are both used, and the table caption contains 'SMPL R epresentation' with a stray space; please standardize throughout.
  4. [Sec. IV-B] The sentence 'We cite results directly from [21]' is vague; please state explicitly which baseline rows were re-run by the authors and which were copied from other papers, including the evaluation protocol used for each.
  5. [Sec. IV-A] The description of the TikTok dataset preparation is terse; please clarify how 3D SMPL parameters were obtained for monocular TikTok videos and how the camera trajectory was derived from GroundingDINO tracks.

Circularity Check

1 steps flagged · score 6.0 of 10

Table V's 'w/o masking' row is the DisCo baseline, so the masking ablation does not isolate the proposed mechanism.

  1. renaming known result [Table V (Sec. IV-C, Masking Mechanism) vs Table II (Sec. IV-B)]
    "TABLE V COMPARISON WITH /WITHOUT MASKING MECHANISM. PSNR SSIM LPIPS w/o 29.03 0.668 0.292 w/ 30.40 0.820 0.200. ... Disco [6] 3.78 29.03 0.668 0.292 59.90 292.80 ... Ours 0.54 30.40 0.820 0.200. ... We cite results directly from [21] for DisCo, MagicAnimate, AnimateAnyone, and Champ, and DreamPose results from [6]."

    Table V's 'w/o' row (29.03 PSNR, 0.668 SSIM, 0.292 LPIPS) is numerically identical to the DisCo baseline row in Table II, and its 'w/' row is identical to the paper's own Ours row. The paper explicitly states DisCo results are cited from prior work rather than rerun. DisCo differs from CFSynthesis in pose representation (DWPose vs textured SMPL), background encoding, and network design, so it is not 'CFSynthesis without masking.' The table therefore renames a known published baseline as the 'w/o masking' condition; the quantitative benefit attributed to the masking mechanism reduces to a relabeled external comparison rather than a controlled ablation.

full rationale

Most of the paper is a standard empirical systems paper: the textured-SMPL representation is a conditioning signal, not a derivation of the output, and the method is evaluated against external benchmarks and baselines. The one self-citation (an earlier NeRF paper with a co-author overlap) appears only in Related Work and is not load-bearing. The central SOTA claim has independent content in Tables II-IV and the qualitative comparisons. However, the sole quantitative evidence for the foreground-background masking mechanism, Table V, is invalid as an ablation: its 'w/o' row exactly reproduces the DisCo baseline in Table II, which the paper says it cites rather than runs, and its 'w/' row exactly matches the paper's own full-model row. DisCo differs from the full model in multiple components, so the comparison conflates the masking mechanism with the whole method change. This reduces the masking contribution's quantitative support to a relabeled baseline (a renaming of a known result), making the paper partially circular in its evidence for one of its two headline designs. The score of 6 reflects that partial circularity; the rest of the paper is not circular.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The framework rests on four external assumptions: pretrained diffusion priors, the SMPL body model, SMPLitex single-image texture estimation, and automatic segmentation/matting tools. The only hand-chosen numeric parameter explicitly introduced is lambda=1 in Eq. (7). No new physical entities are posited.

free parameters (1)
  • lambda in Eq. (7) = 1
    Recomposition weight between background-key and noise-key cross-attention terms; set to 1 without an ablation or sensitivity study, so the value is manually chosen.
assumptions (4)
  • domain assumption Pretrained Stable Diffusion and MusePose weights supply a strong human appearance prior.
    Used as frozen initialization for the denoising U-Net, foreground encoder, and most layers (Sec. III-D); failures of these priors become failures of CFSynthesis.
  • domain assumption Accurate 3D SMPL motion sequences are available for the target character.
    The method requires SMPL poses from 4D-Humans or MotionGPT (Sec. III-B); misalignment or tracking errors directly corrupt the rendered pose signal M^i.
  • ad hoc to paper SMPLitex plus inpainting produces a pseudo-complete UV texture map from one image.
    Eq. (3) and the Limitations paragraph; this single-image texture estimation is the load-bearing step for multi-view appearance.
  • domain assumption GroundingDINO, SGHM, and binarized masks correctly separate foreground from background.
    Used to create training foreground latents and f_seg masks (Sec. IV-A and Eq. (6)); segmentation errors leak into the masked foreground features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CFSynthesis: Controllable and Free-view 3D Human Video Synthesis." pith.science (2026). https://pith.science/paper/OBJGD2ZN

@misc{pith2026241211067,
  author       = {Pith},
  title        = {Pith review of: CFSynthesis: Controllable and Free-view 3D Human Video Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBJGD2ZN}},
  note         = {Machine review of arXiv:2412.11067}
}
read the original abstract

Human video synthesis aims to create lifelike characters in various environments, with wide applications in VR, storytelling, and content creation. While 2D diffusion-based methods have made significant progress, they struggle to generalize to complex 3D poses and varying scene backgrounds. To address these limitations, we introduce CFSynthesis, a novel framework for generating high-quality human videos with customizable attributes, including identity, motion, and scene configurations. Our method leverages a texture-SMPL-based representation to ensure consistent and stable character appearances across free viewpoints. Additionally, we introduce a novel foreground-background separation strategy that effectively decomposes the scene as foreground and background, enabling seamless integration of user-defined backgrounds. Experimental results on multiple datasets show that CFSynthesis not only achieves state-of-the-art performance in complex human animations but also adapts effectively to 3D motions in free-view and user-specified scenarios.

Figures

Figures reproduced from arXiv: 2412.11067 by the authors.

Figure 1
Figure 1. CFSynthesis. Given a single reference image, CFSynthesis can synthesize human videos driven by a texture-based SMPL representation derived from 3D pose estimation or generation. It also integrates user-desired scenes as controllable attributes, enabling the generation of lifelike 3D motion videos with varying backgrounds in free-view. for controllable and free-view 3D human video generation, which has two key design… view at source ↗
Figure 2
Figure 2. An overview of the proposed framework. CFSynthesis first warps an estimated texture map on the given 3D motion sequence and projects it to 2D space through camera pose T i to get the SMPL representation Mi . It is then encoded as pose signals zpose. The foreground and background are separately encoded as zfg and zbg, respectively, and are recomposed during the decoder stage using a masking mechanism. These component… view at source ↗
Figure 3
Figure 3. Implementation of the Masking Mechanism and Pose Extractor. We visualize the operation of the masking mechanism and observe that features in the foreground region tend to diffuse toward the edges and overflow after the first layer of spatial attention. To mitigate this issue, we refine the foreground features using the downsampled f seg l . In the pose extractor, self￾attention effectively captures structured inform… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons between our approach and state-of-the-art methods on the TikTok dataset. We annotate the control conditions in the bottom right corner. The SMPL representation provides robust priors that ensure the best reliability of appearance quality. [14], …
Figure 3
Figure 3. Figure 3: This contour conflict leads to flickering. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison with state-of-the-art methods on the AIST dataset. Our approach demonstrates the best quality in preserving both the fidelity and consistency of character appearance across 360-degree views. This approach guarantees that only pure foreground feat…
Figure 6
Figure 6. Figure 6: Qualitative results on multi-view videos. Our method generates consistent multi-view videos from a single image without exhibiting appearance artifacts. TABLE II QUANTITATIVE RESULTS FOR HUMAN DANCE GENERATION. L1 IS MEASURED IN UNITS OF E-04. DESPITE THE HIGH COST OF …
Figure 7
Figure 7. Figure 7: We directly reference results from the Human4Dit website, which serves as in-the-wild data for CFSynthe￾sis. Leveraging structured prior representations, our method demonstrates improved fidelity to the appearance of images in novel views. For instance, in the case of …
Figure 8
Figure 8. Figure 8: Results on 4D in-the-wild data. CFSynthesis not only achieves human animation but also offers advanced capabilities for human video synthesis, including free-view motion transfer and user-desired scene insertion. Some facial details are missing due to insufficient face…
Figure 9
Figure 9. Figure 9: Ablation results of the SMPL representation. The SMPL repre￾sentation significantly enhances both the geometric accuracy and appearance fidelity of characters, which can be attributed to its 3D representation and the use of texture priors. TABLE V COMPARISON WITH/WITHO…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 35 canonical work pages

  1. [1]

    Magicanimate: Temporally consistent human image animation using diffusion model,

    Z. Xu, J. Zhang, J. H. Liew, H. Yan, J.-W. Liu, C. Zhang, J. Feng, and M. Z. Shou, “Magicanimate: Temporally consistent human image animation using diffusion model,” arXiv preprint arXiv:2311.16498 , 2023

    arXiv 2023

  2. [2]

    Wonder3d: Single image to 3d using cross-domain diffusion,

    X. Long, Y .-C. Guo, C. Lin, Y . Liu, Z. Dou, L. Liu, Y . Ma, S.-H. Zhang, M. Habermann, C. Theobalt et al., “Wonder3d: Single image to 3d using cross-domain diffusion,” arXiv preprint arXiv:2310.15008 , 2023

    arXiv 2023

  3. [3]

    Mvdream: Multi- view diffusion for 3d generation,

    Y . Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang, “Mvdream: Multi- view diffusion for 3d generation,” arXiv preprint arXiv:2308.16512 , 2023

    arXiv 2023

  4. [4]

    Synthesizing moving people with 3d control,

    B. Li, J. Rajasegaran, Y . Gandelsman, A. A. Efros, and J. Ma- lik, “Synthesizing moving people with 3d control,” arXiv preprint arXiv:2401.10889, 2024

    arXiv 2024

  5. [5]

    Text2tex: Text-driven texture synthesis via diffusion models,

    D. Z. Chen, Y . Siddiqui, H.-Y . Lee, S. Tulyakov, and M. Nießner, “Text2tex: Text-driven texture synthesis via diffusion models,” arXiv preprint arXiv:2303.11396, 2023

    arXiv 2023

  6. [6]

    Disco: Disentangled control for referring human dance generation in real world,

    T. Wang, L. Li, K. Lin, C.-C. Lin, Z. Yang, H. Zhang, Z. Liu, and L. Wang, “Disco: Disentangled control for referring human dance generation in real world,” arXiv preprint arXiv:2307.00040 , 2023

    arXiv 2023

  7. [7]

    Hu- mans in 4D: Reconstructing and tracking humans with transformers,

    S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa, and J. Malik, “Hu- mans in 4D: Reconstructing and tracking humans with transformers,” in ICCV, 2023

    2023

  8. [8]

    Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing,

    S. Tsuchida, S. Fukayama, M. Hamasaki, and M. Goto, “Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing,” in Proceedings of the 20th Interna- tional Society for Music Information Retrieval Conference, ISMIR 2019 , Delft, Netherlands, Nov. 2019

    work page 2019

Show all 78 references
  1. [9]

    Segment anything,

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Doll´ar, and R. Girshick, “Segment anything,” arXiv:2304.02643, 2023

    2023 arXiv

  2. [10]

    Grounding dino: Marrying dino with grounded pre- training for open-set object detection,

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu et al., “Grounding dino: Marrying dino with grounded pre- training for open-set object detection,” arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  3. [11]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y . Chen, F. Yanet al., “Grounded sam: Assembling open-world models for diverse visual tasks,” arXiv preprint arXiv:2401.14159 , 2024

    2024 arXiv

  4. [12]

    Dreampose: Fashion image-to-video synthesis via stable diffusion,

    J. Karras, A. Holynski, T.-C. Wang, and I. Kemelmacher-Shlizerman, “Dreampose: Fashion image-to-video synthesis via stable diffusion,” 2023

    2023

  5. [13]

    Robust human matting via semantic guidance,

    X. Chen, Y . Zhu, Y . Li, B. Fu, L. Sun, Y . Shan, and S. Liu, “Robust human matting via semantic guidance,” in Proceedings of the Asian Conference on Computer Vision (ACCV) , 2022

    2022

  6. [14]

    Animate any- one: Consistent and controllable image-to-video synthesis for character animation,

    L. Hu, X. Gao, P. Zhang, K. Sun, B. Zhang, and L. Bo, “Animate any- one: Consistent and controllable image-to-video synthesis for character animation,” arXiv preprint arXiv:2311.17117 , 2023

    2023 arXiv

  7. [15]

    Dreamoving: A human video generation framework based on diffusion models,

    M. Feng, J. Liu, K. Yu, Y . Yao, Z. Hui, X. Guo, X. Lin, H. Xue, C. Shi, X. Li et al., “Dreamoving: A human video generation framework based on diffusion models,” arXiv e-prints, pp. arXiv–2312, 2023

    2023

  8. [16]

    SMPL: A skinned multi-person linear model,

    M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “SMPL: A skinned multi-person linear model,” ACM Trans. Graphics (Proc. SIGGRAPH Asia) , vol. 34, no. 6, pp. 248:1–248:16, Oct. 2015

    2015

  9. [17]

    A good image generator is what you need for high- resolution video synthesis,

    Y . Tian, J. Ren, M. Chai, K. Olszewski, X. Peng, D. N. Metaxas, and S. Tulyakov, “A good image generator is what you need for high- resolution video synthesis,” arXiv preprint arXiv:2104.15069 , 2021

    2021 arXiv

  10. [18]

    One-shot free-view neural talking-head synthesis for video conferencing,

    T.-C. Wang, A. Mallya, and M.-Y . Liu, “One-shot free-view neural talking-head synthesis for video conferencing,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 10 039–10 049

    2021

  11. [19]

    Generative adversarial networks,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,” Communications of the ACM , vol. 63, no. 11, pp. 139–144, 2020

    2020

  12. [21]

    Champ: Controllable and consistent human image animation with 3d parametric guidance,

    S. Zhu, J. L. Chen, Z. Dai, Y . Xu, X. Cao, Y . Yao, H. Zhu, and S. Zhu, “Champ: Controllable and consistent human image animation with 3d parametric guidance,” arXiv preprint arXiv:2403.14781 , 2024

    2024 arXiv

  13. [22]

    Learning high fidelity depths of dressed humans by watching social media dance videos,

    Y . Jafarian and H. S. Park, “Learning high fidelity depths of dressed humans by watching social media dance videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 12 753–12 762

    2021

  14. [23]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international con- ference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 ...

    2015

  15. [24]

    Pirenderer: Controllable portrait image generation via semantic neural rendering,

    Y . Ren, G. Li, Y . Chen, T. H. Li, and S. Liu, “Pirenderer: Controllable portrait image generation via semantic neural rendering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 13 759–13 768

    2021

  16. [25]

    Thin-plate spline motion model for image animation,

    J. Zhao and H. Zhang, “Thin-plate spline motion model for image animation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 3657–3666

    2022

  17. [26]

    Implicit warping for animation with image sets,

    A. Mallya, T.-C. Wang, and M.-Y . Liu, “Implicit warping for animation with image sets,” Advances in Neural Information Processing Systems , vol. 35, pp. 22 438–22 450, 2022

    2022

  18. [27]

    SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image,

    D. Casas and M. Comino-Trinidad, “SMPLitex: A Generative Model and Dataset for 3D Human Texture Estimation from Single Image,” in British Machine Vision Conference (BMVC) , 2023

    2023

  19. [28]

    Exploring dual-task corre- lation for pose guided person image generation,

    P. Zhang, L. Yang, J.-H. Lai, and X. Xie, “Exploring dual-task corre- lation for pose guided person image generation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 7713–7722

    2022

  20. [29]

    ediff-i: Text-to-image diffu- sion models with an ensemble of expert denoisers,

    Y . Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, Q. Zhang, K. Kreis, M. Aittala, T. Aila, S. Laine et al. , “ediff-i: Text-to-image diffu- sion models with an ensemble of expert denoisers,” arXiv preprint arXiv:2211.01324, 2022

    2022 arXiv

  21. [30]

    Glide: Towards photorealistic image gen- eration and editing with text-guided diffusion models,

    A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image gen- eration and editing with text-guided diffusion models,” arXiv preprint arXiv:2112.10741, 2021

    2021 arXiv

  22. [31]

    Hierarchical text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, vol. 1, no. 2, p. 3, 2022

    2022 arXiv

  23. [32]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in neural information processing systems , vol. 35, ...

    2022

  24. [33]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  25. [34]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 3836–3847

    2023

  26. [35]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,

    H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang, “Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models,” arXiv preprint arXiv:2308.06721 , 2023

    2023 arXiv

  27. [36]

    Align your latents: High-resolution video synthesis with latent diffusion models,

    A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 22 563–22 575

    2023

  28. [37]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,

    J. Z. Wu, Y . Ge, X. Wang, S. W. Lei, Y . Gu, Y . Shi, W. Hsu, Y . Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7623–7633

    2023

  29. [38]

    Person image synthesis via denoising diffusion model,

    A. K. Bhunia, S. Khan, H. Cholakkal, R. M. Anwer, J. Laaksonen, M. Shah, and F. S. Khan, “Person image synthesis via denoising diffusion model,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 5968–5976

    2023

  30. [39]

    Conditional image- to-video generation with latent flow diffusion models,

    H. Ni, C. Shi, K. Li, S. X. Huang, and M. R. Min, “Conditional image- to-video generation with latent flow diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 444–18 455. JOURNAL OF LATEX CLASS FILES, VOL. 14, N...

    2023

  31. [40]

    Dreampose: Fashion image-to-video synthesis via stable diffusion,

    J. Karras, A. Holynski, T.-C. Wang, and I. Kemelmacher-Shlizerman, “Dreampose: Fashion image-to-video synthesis via stable diffusion,” in 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, 2023, pp. 22 623–22 633

    2023

  32. [41]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  33. [42]

    Magica- vatar: Multimodal avatar generation and animation,

    J. Zhang, H. Yan, Z. Xu, J. Feng, and J. H. Liew, “Magica- vatar: Multimodal avatar generation and animation,” arXiv preprint arXiv:2308.14748, 2023

    2023 arXiv

  34. [43]

    Self-avatar animation in virtual reality: Impact of motion signals artifacts on the full-body pose reconstruction,

    A. Maiorca, S. A. Ghasemzadeh, T. Ravet, F. Cresson, T. Dutoit, and C. De Vleeschouwer, “Self-avatar animation in virtual reality: Impact of motion signals artifacts on the full-body pose reconstruction,” arXiv preprint arXiv:2404.18628, 2024

    2024 arXiv

  35. [44]

    Fast regis- tration of photorealistic avatars for vr facial animation,

    C. Patel, S. Bai, T.-L. Wang, J. Saragih, and S.-E. Wei, “Fast regis- tration of photorealistic avatars for vr facial animation,” arXiv preprint arXiv:2401.11002, 2024

    2024 arXiv

  36. [45]

    Gazemodiff: Gaze-guided diffusion model for stochastic human motion prediction,

    H. Yan, Z. Hu, S. Schmitt, and A. Bulling, “Gazemodiff: Gaze-guided diffusion model for stochastic human motion prediction,” arXiv preprint arXiv:2312.12090, 2023

    2023 arXiv

  37. [46]

    Llanimation: Llama driven gesture animation,

    J. Windle, I. Matthews, and S. Taylor, “Llanimation: Llama driven gesture animation,” arXiv e-prints, pp. arXiv–2405, 2024

    2024

  38. [47]

    Sparse recon- struction of optical doppler tomography based on state space model,

    Z. Li, J. Ren, W. Cheng, C. Du, Y . Pan, and H. Ling, “Sparse recon- struction of optical doppler tomography based on state space model,” arXiv preprint arXiv:2404.17484 , 2024

    2024 arXiv

  39. [48]

    Laser: Tuning-free llm-driven attention control for efficient text-conditioned image-to-animation,

    H. Zheng, W. Zhang, Y . Wang, H. Zhou, J. Liu, J. Li, Z. Lv, S. Tang, and Y . Zhuang, “Laser: Tuning-free llm-driven attention control for efficient text-conditioned image-to-animation,” arXiv preprint arXiv:2404.13558, 2024

    2024 arXiv

  40. [49]

    Maskel: A model for human whole-body x-rays generation from human masking images,

    Y . Xi, B. Cheng, J. Cai, J. J. Zhang, and X. Yang, “Maskel: A model for human whole-body x-rays generation from human masking images,” arXiv preprint arXiv:2404.09000 , 2024

    2024 arXiv

  41. [50]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning,

    Y . Guo, C. Yang, A. Rao, Z. Liang, Y . Wang, Y . Qiao, M. Agrawala, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to- image diffusion models without specific tuning,” International Confer- ence on Learning Representations , 2024

    2024

  42. [51]

    Sparsectrl: Adding sparse controls to text-to-video diffusion models,

    Y . Guo, C. Yang, A. Rao, M. Agrawala, D. Lin, and B. Dai, “Sparsectrl: Adding sparse controls to text-to-video diffusion models,” arXiv preprint arXiv:2311.16933, 2023

    2023 arXiv

  43. [52]

    Effective whole-body pose estimation with two-stages distillation,

    Z. Yang, A. Zeng, C. Yuan, and Y . Li, “Effective whole-body pose estimation with two-stages distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4210–4220

    2023

  44. [53]

    Image quality assessment: from error visibility to structural similarity,

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing , vol. 13, no. 4, pp. 600–612, 2004

    2004

  45. [54]

    Towards accurate generative models of video: A new metric & challenges,

    T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, “Towards accurate generative models of video: A new metric & challenges,” arXiv preprint arXiv:1812.01717 , 2018

    2018 arXiv

  46. [55]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018

    2018

  47. [56]

    On loss functions for deep neural networks in classification,

    K. Janocha and W. M. Czarnecki, “On loss functions for deep neural networks in classification,” arXiv preprint arXiv:1702.05659 , 2017

    2017 arXiv

  48. [57]

    Human4dit: 360-degree human video generation with 4d diffusion transformer,

    R. Shao, Y . Pang, Z. Zheng, J. Sun, and Y . Liu, “Human4dit: 360-degree human video generation with 4d diffusion transformer,”

  49. [58]

    Nerf: Representing scenes as neural radiance fields for view synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM , vol. 65, no. 1, pp. 99–106, 2021

    2021

  50. [59]

    3d gaussian splatting for real-time radiance field rendering

    B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph. , vol. 42, no. 4, pp. 139–1, 2023

    2023

  51. [60]

    Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians,

    L. Hu, H. Zhang, Y . Zhang, B. Zhou, B. Liu, S. Zhang, and L. Nie, “Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 634– 644

    2024

  52. [61]

    Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling,

    Z. Li, Z. Zheng, L. Wang, and Y . Liu, “Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 19 711–19 722

    2024

  53. [62]

    Neural actor: Neural free-view synthesis of human actors with pose control,

    L. Liu, M. Habermann, V . Rudnev, K. Sarkar, J. Gu, and C. Theobalt, “Neural actor: Neural free-view synthesis of human actors with pose control,” ACM transactions on graphics (TOG) , vol. 40, no. 6, pp. 1– 16, 2021

    2021

  54. [63]

    Animatable neural radiance fields for modeling dynamic human bodies,

    S. Peng, J. Dong, Q. Wang, S. Zhang, Q. Shuai, X. Zhou, and H. Bao, “Animatable neural radiance fields for modeling dynamic human bodies,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14 314–14 323

    2021

  55. [64]

    Humannerf: Free-viewpoint rendering of moving people from monocular video,

    C.-Y . Weng, B. Curless, P. P. Srinivasan, J. T. Barron, and I. Kemelmacher-Shlizerman, “Humannerf: Free-viewpoint rendering of moving people from monocular video,” in Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition , 2022, pp. 16 210–16 220

    2022

  56. [65]

    Tech: Text-guided reconstruction of lifelike clothed humans,

    Y . Huang, H. Yi, Y . Xiu, T. Liao, J. Tang, D. Cai, and J. Thies, “Tech: Text-guided reconstruction of lifelike clothed humans,” in 2024 International Conference on 3D Vision (3DV) . IEEE, 2024, pp. 1531– 1542

    2024

  57. [66]

    Arch: Animatable reconstruction of clothed humans,

    Z. Huang, Y . Xu, C. Lassner, H. Li, and T. Tung, “Arch: Animatable reconstruction of clothed humans,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 3093–3102

    2020

  58. [67]

    Tada! text to animatable digital avatars,

    T. Liao, H. Yi, Y . Xiu, J. Tang, Y . Huang, J. Thies, and M. J. Black, “Tada! text to animatable digital avatars,” in 2024 International Conference on 3D Vision (3DV) . IEEE, 2024, pp. 1508–1519

    2024

  59. [68]

    En3d: An enhanced generative model for sculpting 3d humans from 2d synthetic data,

    Y . Men, B. Lei, Y . Yao, M. Cui, Z. Lian, and X. Xie, “En3d: An enhanced generative model for sculpting 3d humans from 2d synthetic data,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9981–9991

    2024

  60. [69]

    Motiongpt: Human motion as a foreign language,

    B. Jiang, X. Chen, W. Liu, J. Yu, G. Yu, and T. Chen, “Motiongpt: Human motion as a foreign language,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  61. [70]

    Unianimate: Taming unified video diffusion models for consistent human image animation,

    X. Wang, S. Zhang, C. Gao, J. Wang, X. Zhou, Y . Zhang, L. Yan, and N. Sang, “Unianimate: Taming unified video diffusion models for consistent human image animation,” arXiv preprint arXiv:2406.01188 , 2024

    2024 arXiv

  62. [71]

    Motionfollower: Editing video motion via lightweight score-guided diffusion,

    S. Tu, Q. Dai, Z. Zhang, S. Xie, Z.-Q. Cheng, C. Luo, X. Han, Z. Wu, and Y .-G. Jiang, “Motionfollower: Editing video motion via lightweight score-guided diffusion,” arXiv preprint arXiv:2405.20325 , 2024

    2024 arXiv

  63. [72]

    Vividpose: Advancing stable video diffusion for realistic human image animation,

    Q. Wang, Z. Jiang, C. Xu, J. Zhang, Y . Wang, X. Zhang, Y . Cao, W. Cao, C. Wang, and Y . Fu, “Vividpose: Advancing stable video diffusion for realistic human image animation,” arXiv preprint arXiv:2405.18156 , 2024

    2024 arXiv

  64. [73]

    Musepose: a pose-driven image-to-video framework for virtual human generation,

    Z. Tong, C. Li, Z. Chen, B. Wu, and W. Zhou, “Musepose: a pose-driven image-to-video framework for virtual human generation,” arxiv, 2024

    2024

  65. [74]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), June 2022, pp. 10 684–10 695

    2022

  66. [75]

    Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion,

    D. Chang, Y . Shi, Q. Gao, H. Xu, J. Fu, G. Song, Q. Yan, Y . Zhu, X. Yang, and M. Soleymani, “Magicpose: Realistic human poses and facial expressions retargeting with identity-aware diffusion,” in Forty- first International Conference on Machine Learning , 2023

    2023

  67. [76]

    Mocogan: Decomposing motion and content for video generation,

    S. Tulyakov, M.-Y . Liu, X. Yang, and J. Kautz, “Mocogan: Decomposing motion and content for video generation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 1526– 1535

    2018

  68. [77]

    Video-to-video synthesis,

    T.-C. Wang, M.-Y . Liu, J.-Y . Zhu, G. Liu, A. Tao, J. Kautz, and B. Catan- zaro, “Video-to-video synthesis,” in Advances in Neural Information Processing Systems (NeurIPS) , 2018

    2018

  69. [78]

    Follow your pose: Pose-guided text-to-video generation using pose-free videos,

    Y . Ma, Y . He, X. Cun, X. Wang, Y . Shan, X. Li, and Q. Chen, “Follow your pose: Pose-guided text-to-video generation using pose-free videos,” arXiv preprint arXiv:2304.01186 , 2023

    2023 arXiv

  70. [2024]

    Available: https://arxiv.org/abs/2405.17405

    [Online]. Available: https://arxiv.org/abs/2405.17405

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.