{"id":"bd2bdd1d-fdcd-4864-ae5c-78b55f4f0878","arxiv_id":"2504.13021","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A self-supervised StyleGAN2 method transfers pose and expression from a source face onto a target identity with near-real-time inference.","lead":"Researchers built a system that copies the head pose and facial expression from one person's photo onto another person's identity, generating a new face image in under a second. It uses the well-known StyleGAN2 generator with two separate encoders and learns without manual labels from videos of many people.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Default training objective never supervises cross-identity pose/expression transfer; success depends entirely on an unverified disentanglement of the fixed ReStyle identity encoder.","rationale":"The central claim is an empirical one, and the paper provides quantitative support in Table 1, so I do not call it false. The load-bearing issue is that the default training objective (Sec. 3.2 and Sec. 3.4) contains no loss that supervises cross-identity pose/expression transfer: pixel and LPIPS losses are defined for the same-identity pair only, the identity loss is the sole cross-identity loss, and the CosFace loss that would align source and output motion embeddings is disabled (w_cos=0). Thus the reported transfer must arise from the composition of Em(s) and the fixed ReStyle pSp Ei(t) through a single linear layer. That composition is reliable only if Ei carries no pose/expression information; the paper does not test this. The reader's weakest assumption is essentially the same, but I sharpen it to a training-signal gap. The paper's self-reported limitations—fully closed eyes (Sec. 5) and the proxy nature of the facial-landmark expression metric (Sec. 4.3)—weaken the strength of the claim but do not by themselves invalidate it. A single behavioral test, varying the target frame's pose/expression with the source fixed, would settle the disentanglement question. If the test shows output invariance to target pose/expression, the conditional accept is justified; if not, the central claim's training basis is missing. Since no code is released, the test requires reimplementation, which is feasible from the description. On this basis I leave the reader's CONDITIONAL verdict unchanged.","tokens_in":10274,"tokens_out":10558,"duration_ms":97962,"concrete_test":"Behavioral leakage test: fix one source s with a distinctive pose/expression. For 20 target identities, take two frames t1 and t2 of the same identity with markedly different yaw/pitch/expression (e.g., at least 15 degrees yaw difference). Generate g_{s→t1} and g_{s→t2} with the released or retrained default model. Measure pose error and facial-landmark aspect-ratio correlation between the two outputs. If the outputs differ systematically (pose MAE greater than 2 degrees or FL correlation greater than 0.3 across the set), Ei leaks target pose/expression into the latent and the disentanglement assumption fails; if the outputs are statistically indistinguishable, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the default (best) configuration, Sec. 3.2 defines pixel (Eq. 1) and LPIPS (Eq. 2) losses only for the same-identity pair g_{sA→tA}. The only loss applied to the cross-identity output g_{sA→tB} is the ArcFace identity loss (Eq. 3). The CosFace loss (Eqs. 4–5), designed to align the motion embeddings of the output with the source, is disabled in the best model (w_cos=0, Sec. 3.4). Consequently, the central claim—that pose and expression of the source are transferred onto the target identity—is not directly supervised at training time. The reported transfer must emerge from composing Em(s) with the fixed ReStyle pSp embedding Ei(t) through a single linear layer M. This requires Ei(t) to be free of pose/expression information; otherwise the cross-identity output can satisfy the identity loss while retaining the target's own pose/expression. The paper provides no analysis, ablation, or control for this. The same-identity reconstruction losses cannot by themselves enforce cross-identity transfer, since a solution that simply reconstructs tB for cross-identity inputs would also satisfy Eq. 3. Thus the central claim rests on an unverified architectural assumption, not on an explicit training signal.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a one-shot face reenactment method that transfers pose and expression from a source face image onto the identity of a target face image. The architecture consists of a motion encoder (ResNet-IR SE50), a fixed identity encoder (ReStyle pSp), a single linear mapping network, and a fixed pre-trained StyleGAN2 generator. Em and M are trained self-supervised on unlabeled VoxCeleb2 video sequences using a composite loss: L2 and LPIPS losses on same-identity reconstruction outputs, an ArcFace identity loss on cross-identity outputs, an optional CosFace motion-alignment loss that is disabled in the best configuration, and cropped variants of the reconstruction losses. Experiments on a VoxCeleb2 test split report pose MAE of 7.67 degrees, expression landmark cross-correlation of 0.62, and identity cosine similarity of 0.80, with two latent-space-editing baselines. The paper also describes a procedure for transferring pose/expression to randomly sampled StyleGAN2 identities and reports close-to-real-time inference after a one-time identity encoding step.","tokens_in":10541,"tokens_out":7117,"duration_ms":67083,"significance":"If the claims hold, the main contribution is a simple, fast, and reproducible pipeline built from open-source pretrained components that demonstrates how a linear fusion of separate motion and identity embeddings can drive StyleGAN2 for pose/expression transfer. The self-supervised video-based training, the use of standard components, and the reported inference speed are genuine strengths. However, the quantitative evidence is not yet conclusive: the cross-identity output is never directly supervised for pose and expression, the quantitative evaluation lacks statistical validation and a strong single-image baseline, and the expression metric is a coarse proxy. The paper is a plausible workshop-level result, but for a journal version the evaluation and the architectural justification need substantial reinforcement.","major_comments":[{"comment":"The cross-identity output g_{sA->tB} is supervised only by the ArcFace identity loss L_ID (Eq. 3); the L2 and LPIPS losses and their cropped variants are defined only for the same-identity output g_{sA->tA}, and the CosFace loss is disabled in the best model (w_cos = 0). Consequently, the central claim that the source pose and expression are transferred onto the target identity is not directly trained. The method instead relies on the untested assumption that Ei(t) is free of pose/expression information and that a single linear layer on Em(s) concatenated with Ei(t) yields a valid latent code. This assumption is load-bearing: if Ei(t) leaks pose or expression, the cross-identity output can satisfy Eq. (3) while retaining the target's own pose and expression, and the reported pose/expression transfer would be an artifact of the evaluation setup. The authors should provide a disentanglement analysis, e.g., train a linear probe on Ei(t) to predict yaw/pitch and expression, and should ablate by adding a mild reconstruction or LPIPS loss on the cross-identity output or by engaging the CosFace loss, reporting the effect on all Table 1 metrics. Without such evidence, the mechanism behind the transfer remains unjustified.","section":"Sec. 3.2 and Sec. 3.4, Eqs. (1)-(6)"},{"comment":"The quantitative comparison does not report error bars, confidence intervals, significance tests, or the number of evaluated frames/videos. The differences among methods are small in absolute terms (pose MAE: 7.673 vs. 7.968 vs. 8.491; expression FL CORR: 0.620 vs. 0.656 for Base pSp), and the expression metric is actually lower for the proposed method than for Base pSp. Without per-video standard errors and a paired significance test over the 120 test identities, the claim that the proposed method improves pose transfer over the baselines is not statistically supported. The authors should report the mean and standard error over videos (or bootstrap confidence intervals) and conduct a paired test (e.g., Wilcoxon signed-rank) between the methods.","section":"Table 1 and Sec. 4.3"},{"comment":"The baseline comparison is not a single-image reenactment comparison: the baseline does not use the actual target image but a frontalized, average-expression reconstruction computed from a video of the target identity (via vertical flipping and averaging latent codes). This gives the baseline two disadvantages unrelated to the core task: it requires a video rather than a single image, and the averaged frontal frame may itself be a poor proxy for the target's identity and expression. The comparison therefore does not isolate the benefit of the proposed method over latent-space editing for the stated one-shot setting. The authors should either replace the baseline with a true single-image latent-editing baseline (e.g., applying the edit vector computed from a matched source frame pair directly to the target latent code) or clearly delimit the claim to the video-based latent-arithmetic setting and explain why the baseline is appropriate.","section":"Sec. 4.1"},{"comment":"The FL statistic is a proxy that computes cross-correlations of aspect ratios of eyes, mouth, and eyebrows, and the paper itself acknowledges it cannot capture asymmetric expressions, eyeball movements, or many subtle expression changes. The reported value of 0.62 is not anchored to human perception or to a known-good reference, so it is unclear what level of correlation constitutes successful expression transfer. The authors should supplement this with a user study, a facial action unit detector, or an expression classification metric to substantiate the expression-transfer claim, or at least calibrate the metric on a trivial baseline (e.g., correlation between the source and an identity-preserving but expression-frozen reconstruction).","section":"Sec. 4.3, expression fidelity metric"},{"comment":"The identity metric reported in Table 1 is the ArcFace cosine similarity, and the training loss L_ID in Eq. (3) is exactly its complement. The ID CSIM of 0.80 therefore partly reflects direct optimization of the evaluation metric, not an independent verification of identity preservation. The authors should explicitly acknowledge this circularity or report a second identity metric based on a different face recognition model to strengthen the claim.","section":"Sec. 4.3, ID CSIM"}],"minor_comments":[{"comment":"The paper states in Sec. 3.1 that input images are 'aligned and resized to 256 x 256 pixels,' but Sec. 3.3 says the official StyleGAN preprocessing script normalizes images to 224 x 224 pixels. This inconsistency should be resolved or clarified.","section":"Sec. 3.3 vs. Sec. 3.1"},{"comment":"The CosFace loss is typeset in a way that is hard to read: the expression 'd(zi,z j)' and the fraction in Eq. (5) would benefit from standard mathematical notation and consistent subscript spacing.","section":"Sec. 3.2, Eqs. (4)-(5)"},{"comment":"There is a typo: 'In praticular' should be 'In particular'.","section":"Sec. 3.2"},{"comment":"The description of the facial-landmark aspect ratios and the cross-correlation computation is vague; please provide the exact formulas and state whether the cross-correlation is normalized (e.g., Pearson correlation) and how the reported average is computed over videos.","section":"Sec. 4.3, FL statistic"},{"comment":"The paper should state the number of test videos, frames, and source-target pairs used for the numbers in Table 1, and whether the metrics are averaged over frames, videos, or identities.","section":"Sec. 4.3, dataset size"},{"comment":"The related work discusses X2Face [33] and MegaPortraits [9], but the experiments include no comparison with these methods; a sentence explaining why they are not compared (e.g., no public code, different input requirements) would improve the framing.","section":"Sec. 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a camera-ready version of a CVWW 2024 workshop paper. The core idea is simple and potentially useful, but the journal version would need a substantially stronger evaluation: error bars and significance tests, a fair single-image baseline, and an analysis of the disentanglement assumption. The issues raised in the major comments are fixable within the scope of the paper, so I do not recommend rejection, but the current evidence does not yet support the paper's central claims at journal level."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The stress-test note is right, and it matters. In the default model (w_cos=0), the L2 and LPIPS losses are only computed for same-identity reconstructions (sA->tA), and the only loss on the cross-identity output is ArcFace identity preservation. No term pushes the output's pose or expression toward the source. The model could satisfy training by reconstructing the target's own motion as long as ArcFace stays high. The paper offers no analysis, ablation, or control showing that the fixed ReStyle/pSp identity embedding is free of pose/expression information. That is the load-bearing gap behind the central claim.\n\nThat said, the paper has real merits. The proposed pipeline—a trainable motion encoder plus a fixed identity encoder feeding a single linear mapping into fixed StyleGAN2—is a clean, simple extension of prior work (hairstyle transfer, expression-only transfer) to one-shot pose+expression transfer with near-real-time inference. Self-supervised training on VoxCeleb2 without labels is sensible, and the authors are honest about their baseline's limitations and about the closed-eyes failure case. The expression proxy using landmark aspect-ratio correlation over videos is thoughtful, and Table 1 at least compares variants and two baselines.\n\nThe soft spots are in proportion. The evaluation has no comparison to Megaportraits, X2Face, or first-order-motion methods; the baseline is deliberately weak and requires video; there are no error bars or significance tests; and ArcFace is both a training loss and the identity metric, which is mild circularity. Also, the expression metric shows the baseline pSp actually beats the proposed method (0.656 vs 0.620), so the claimed advantage really reduces to pose and identity. The code and model are promised but not released, which makes verification harder.\n\nI do not think the method is fake or incoherent—the qualitative results suggest it often does transfer motion. But as written, the paper does not establish that the transfer is caused by the intended mechanism rather than by target-motion leakage through the identity encoder. That is a fixable problem: add an explicit source-motion loss on cross-identity outputs, or an ablation where target pose/expression is held fixed while source varies, or at minimum a study of how much pose/expression the identity embedding retains.\n\nFor a reading group, this is a useful case study in how a self-supervised objective and a method's stated claim can drift apart. I would not cite it as a reliable method until the supervision gap is closed and the code appears, but I would still send it to a serious referee: a good reviewer could either extract the needed control experiments or help the authors see why the current evidence is insufficient.","headline":"The architecture is simple and the write-up is honest, but in the best configuration nothing directly supervises cross-identity pose/expression transfer, so the central claim rests on an unverified disentanglement assumption.","tokens_in":11078,"tokens_out":3207,"would_cite":false,"duration_ms":36447,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Given a source and a target face portrait, this paper claims a single linear mapping in StyleGAN2's latent space can transfer the source's pose and expression onto the target's identity, trained without labels.","keywords":["face reenactment","pose transfer","expression transfer","StyleGAN2","latent space manipulation","self-supervised learning","image-to-image translation","video-driven animation"],"falsifier":"Take a video of one person, feed each frame through the fixed identity encoder, and train a linear classifier or regressor to predict head pose (yaw/pitch) from the resulting identity embeddings; if pose can be predicted well above chance from those embeddings, the claimed identity-motion separation fails, and the linear mapping cannot be expected to transfer pose independently across identities.","tokens_in":10051,"feed_emoji":"🎭","tokens_out":8604,"duration_ms":80688,"temperature":0.7,"pith_summary":"This paper proposes a method that takes two face portraits, a source and a target, and outputs a new image of the target person wearing the source's pose and facial expression. The authors argue that the task can be solved without 3D models, manual labels, or training a new generator: a motion encoder and a fixed identity encoder feed concatenated embeddings through one linear mapping into the latent space of a pretrained StyleGAN2, which renders the result. Training is self-supervised on unlabeled video clips of many identities, using reconstruction, perceptual, and face-recognition losses, plus an optional contrastive loss on motion embeddings. The reported numbers put identity preservation at 0.80 cosine similarity, close to the 0.83 ceiling of inverting the target image alone, which suggests the transfer step itself costs little identity fidelity. If the method works, it makes photorealistic face reenactment from a single still image practical, since the driving video can be rendered at about 20 frames per second.","feed_headline":"Swap pose and expression between faces in near real time","feed_subtitle":"A self-supervised StyleGAN pipeline copies a source's motion onto a target's face, holding identity close to the encoder's ceiling.","key_machinery":"The load-bearing component is the mapping network $M$: a single fully connected linear layer that takes the concatenation of the motion embedding $E_m(s)$ and the identity embedding $E_i(t)$, both living in the extended latent space $W^+$ of dimension $18\\times 512$, and produces the StyleGAN2 latent code that generates the output. Because the generator $G$ and the identity encoder $E_i$ are frozen, the entire learning problem reduces to finding a linear projection that mixes motion and identity without confusing them; the motion encoder $E_m$ is the only other trained part. The identity encoder is a pretrained ReStyle iteration of the pSp encoder, chosen after the authors found the alternative e4e configuration struggled to preserve identity. A practical detail is that the L2 and LPIPS losses are computed on a central $188\\times 188$ crop, which the authors report was important for faithful expression transfer, since the surrounding hair and background otherwise distracted training.","core_discovery":"The central claim is that pose and expression form an identity-independent motion code that can be extracted from one face and re-injected into another face's identity code, all inside StyleGAN2's latent space, without any explicit 3D face model. Formally, the system computes $g_{s\\rightarrow t}=G(M(E_m(s)\\oplus E_i(t)))$, where $E_m$ embeds the source's motion, $E_i$ embeds the target's identity with a fixed pretrained encoder, the linear mapping $M$ mixes the two, and the fixed generator $G$ renders the image. The motion encoder and mapping are trained on VoxCeleb2 video clips by reconstructing same-identity pairs with L2 and LPIPS losses applied to a central crop, by enforcing identity preservation with ArcFace cosine similarity, and optionally by a CosFace contrastive loss that pulls together motion embeddings of the same expression and pushes apart different ones. The final model reports mean absolute pose error of 7.67 on yaw and pitch, facial-landmark expression correlation of 0.62, and identity cosine similarity of 0.80, beating the latent-arithmetic baselines on pose and identity. The authors also show the same pipeline can impose a source's pose and expression onto randomly generated StyleGAN identities, which makes the motion code usable as a control signal for synthesizing new faces.","pith_inferences":["If the identity code from the pretrained encoder is genuinely free of pose and expression, the motion code could be driven by non-image inputs such as audio or body landmarks, since the mapping accepts any vector in the motion-embedding space; the paper only demonstrates image-driven motion.","The linearity of the mapping suggests that simple vector arithmetic on motion embeddings, such as averaging several source expressions or interpolating between poses, would transfer cleanly across identities; the paper does not test this, but it follows from the architecture.","The paper's expression metric is a proxy based on landmark aspect ratios, so it may understate the method on asymmetric expressions or eye movements; a FAU-based evaluation on data with strong activations could reveal where the proxy misleads.","Because random identities must be rendered and then inverted to enter the model's operating subspace, applications that need arbitrary StyleGAN latent codes would have to round-trip through the encoder, suggesting the mapping lives in a restricted subspace of $W^+$."],"forward_implications":["A single target still plus any driving video yields a reenacted video: after one half-second inversion of the target, the model renders up to 20 frames per second.","Identity similarity of 0.80 versus a 0.83 inversion ceiling means the transfer step costs almost no identity fidelity, so the encoder is the bottleneck rather than the mapping.","Because the generator is frozen, the method inherits StyleGAN2's coverage; expressions the generator cannot render, such as fully closed eyes, will fail until the generator is fine-tuned or regularized.","The method beats both ReStyle-based latent-arithmetic baselines on pose error and identity preservation, so the proposed mapping is a more reliable way to reenact faces than direct vector arithmetic in the latent space.","Random StyleGAN identities can be driven by the same motion code, so pose and expression control extends beyond real photographs to synthetic faces."],"supporting_citations":[{"why":"It supplies the fixed StyleGAN2 generator that renders the final image from the mapped $W^+$ latent code.","marker":"[14]"},{"why":"It provides the ReStyle iterative-refinement encoder used as the identity encoder $E_i$ and for inverting inputs.","marker":"[4]"},{"why":"It defines the pSp encoder configuration that the identity encoder is based on and that the baseline uses.","marker":"[20]"},{"why":"It supplies the VoxCeleb2 video corpus of over six thousand identities used for self-supervised training and evaluation.","marker":"[6]"},{"why":"It provides ArcFace embeddings for the identity loss during training and for identity-similarity evaluation.","marker":"[8]"},{"why":"It defines the LPIPS perceptual loss used in training to match generated images to source appearance.","marker":"[37]"},{"why":"It defines the CosFace loss used in the Ours-Cos variant to align motion embeddings of matching poses and expressions.","marker":"[31]"},{"why":"It provides the CNN head-pose estimator used to measure yaw and pitch error in the quantitative evaluation.","marker":"[22]"},{"why":"It supplies the Dlib facial landmark detector used to compute the expression-transfer correlation metric.","marker":"[15]"},{"why":"It motivates the CosFace loss usage and the one-shot cross-reenactment setting the paper compares against.","marker":"[9]"}],"fun_headline_variants":["Self-supervised StyleGAN swaps face pose and expression","Copy one face's motion onto another identity in near real time","StyleGAN transfers pose and expression without manual labels","Random identities get controllable pose and expression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the pretrained identity encoder produces an embedding that contains identity only, cleanly separated from pose and expression, so a single linear mapping over the concatenated codes can swap motion without disturbing identity; the paper does not measure how entangled the identity code actually is.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised StyleGAN swaps face pose and expression","Copy one face's motion onto another identity in near real time","StyleGAN transfers pose and expression without manual labels","Random identities get controllable pose and expression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001036,"raw_usage":{"total_tokens":4346,"prompt_tokens":913,"completion_tokens":3433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":3372}},"tokens_in":529,"tokens_out":3433,"duration_ms":25422,"temperature":1.0,"reasoning_tokens":3372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:16:00.744871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a video of one person, feed each frame through the fixed identity encoder, and train a linear classifier or regressor to predict head pose (yaw/pitch) from the resulting identity embeddings; if pose can be predicted well above chance from those embeddings, the claimed identity-motion separation fails, and the linear mapping cannot be expected to transfer pose independently across identities.","supporting_citations":[{"cited_title":"Karras, S","cited_arxiv_id":null,"evidence_quote":"It supplies the fixed StyleGAN2 generator that renders the final image from the mapped $W^+$ latent code."},{"cited_title":"Alaluf, O","cited_arxiv_id":null,"evidence_quote":"It provides the ReStyle iterative-refinement encoder used as the identity encoder $E_i$ and for inverting inputs."},{"cited_title":"Richardson, Y","cited_arxiv_id":null,"evidence_quote":"It defines the pSp encoder configuration that the identity encoder is based on and that the baseline uses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the VoxCeleb2 video corpus of over six thousand identities used for self-supervised training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides ArcFace embeddings for the identity loss during training and for identity-similarity evaluation."},{"cited_title":"Zhang, P","cited_arxiv_id":null,"evidence_quote":"It defines the LPIPS perceptual loss used in training to match generated images to source appearance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines the CosFace loss used in the Ours-Cos variant to align motion embeddings of matching poses and expressions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the CNN head-pose estimator used to measure yaw and pitch error in the quantitative evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the Dlib facial landmark detector used to compute the expression-transfer correlation metric."},{"cited_title":"MegaPortraits: One-shot Megapixel Neural Head Avatars","cited_arxiv_id":"2207.07621","evidence_quote":"It motivates the CosFace loss usage and the one-shot cross-reenactment setting the paper compares against."}],"review_version":1}