{"id":"aec779fb-92d5-48ee-b406-3bb69f581603","arxiv_id":"2412.07333","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FPDM uses contrastively learned fusion embeddings of source appearance and target pose as conditioning for a diffusion model, achieving top scores on several pose-guided person synthesis metrics.","lead":"This paper introduces FPDM, a two-stage diffusion model that first learns a \"fusion embedding\" aligning a person's source image and target pose with the target image, then uses that embedding to condition image generation. The approach improves texture fidelity and pose consistency on person synthesis benchmarks, with potential uses in virtual try-on and sign language video generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of explicit fusion embedding alignment rests on ablation deltas (B6 vs B4/B5 in Table 3) that are tiny and unreplicated; without error bars or significance tests, 'substantially improves' is not established.","rationale":"The reader's weakest assumption concerns the information content of the three-channel RGB pose image. That is a legitimate design risk, but it is less load-bearing than the ablation gap: the strong benchmark numbers themselves indirectly suggest the pose representation carries enough signal, and adding a quantitative comparison to multi-channel pose encodings would be useful without necessarily overturning the central mechanism. The ablation evidence is the direct support for the paper's stated contribution, which is that explicit fusion embedding alignment improves generation fidelity and consistency. If the B6 improvements over B4 and B5 are within noise, the paper overclaims its central result. I therefore agree with the reader's overall conditional verdict, but for a somewhat different reason. The Phoenix hand-detector mismatch and the author-list inconsistency are secondary concerns. No ad hominem is implied; the issue is that the reported numbers do not yet substantiate the headline claim. A reproducibility check with multiple seeds and confidence intervals would settle whether the mechanism actually matters or is a marginal add-on to the source-image encoder and pose encoder.","tokens_in":12817,"tokens_out":4095,"duration_ms":44234,"concrete_test":"Rerun B4, B5, and B6 with at least 3 random seeds using identical hyperparameters, evaluate on the same DeepFashion test split, and report paired 95% bootstrap confidence intervals for LPIPS, SSIM, PSNR, and FID, plus a paired significance test. If B6 does not beat B4 and B5 by more than the interval width on perceptual or pixel metrics, the 'substantial improvement' claim should be removed or softened; if it does, the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and conclusion claim that ablation studies confirm explicit fusion embedding alignment substantially improves texture fidelity and consistency. The only quantitative support is Table 3. B6 (Source-Enhanced Pose Fusion IPF) versus B3 (no IPF, DINOv2-512): LPIPS 0.1717 vs 0.1737, SSIM 0.7487 vs 0.7470, PSNR 18.1969 vs 18.1254. Against B4 (CLIP image embedding, no IPF): LPIPS 0.1717 vs 0.1734, SSIM 0.7487 vs 0.7466, PSNR 18.1969 vs 18.1315, while FID is worse (7.534 vs 7.464). B5 (baseline fusion) is statistically indistinguishable from B6. The paper itself concedes the differences are 'noticeable but not substantial.' No random seeds, error bars, or significance tests are reported, and the FID direction is opposite to the claim. Since the central novelty is precisely that explicit alignment yields better generation, the evidence does not support 'substantially improves.' The first-stage retrieval result (A3 R@1 = 0.997) shows embedding-space alignment, but retrieval accuracy does not by itself establish a generation benefit; the second-stage ablation is the missing link.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FPDM, a two-stage framework for pose-guided person image synthesis. In the first stage, CLIP image encoders and a Combiner module fuse the source image embedding and a target pose image embedding into a single fusion embedding, trained with a source-enhanced InfoNCE contrastive loss to align with the target image embedding. In the second stage, a Stable Diffusion backbone is conditioned on DINOv2 source-image features, a four-layer CNN pose encoding, and the frozen fusion embedding, with classifier-free guidance applied jointly over appearance, pose, and fusion conditions. The method is evaluated on DeepFashion and RWTH-PHOENIX-Weather 2014T, reporting state-of-the-art LPIPS, SSIM, and PSNR on DeepFashion and large improvements on Phoenix, along with ablations of the fusion module and pose/source encoders.","tokens_in":13015,"tokens_out":4751,"duration_ms":47961,"significance":"If the reported results are reproducible, the paper offers a practically useful conditioning mechanism for diffusion-based pose-guided person image synthesis. The authors deserve credit for releasing code, for introducing FIDt to address the known DeepFashion train/test distribution shift, and for reporting a first-stage retrieval result (R@1 = 0.997) that independently demonstrates embedding-space alignment. The Phoenix sign-language evaluation is also a valuable extension. However, the central causal claim that explicit fusion embedding alignment 'substantially improves' generation is not supported by the second-stage ablation, whose deltas are small, unreplicated, and partly reverse in FID; the Phoenix hand-metric comparison is weakened by a detector mismatch; and the assertion that a three-channel RGB pose image suffices is only visually verified, with no quantitative pose-representation ablation. These are load-bearing issues for the manuscript's stated novelty claims.","major_comments":[{"comment":"The Hand PE comparison in Table 4 is not apples-to-apples: SignGAN's Hand PE is computed with the hand pose detector of Ge et al., while FPDM uses HRNetv2. Hand PE is a keypoint-distance metric and is known to be sensitive to the detector used to extract the keypoints. The authors should recompute at least one of the two methods with the other detector and report both values, or otherwise state clearly that the Hand PE numbers are not directly comparable. This matters because the Phoenix table is one of the two main empirical pillars of the paper.","section":"Quantitative Results for Sign Language Image Generation (Table 4)"},{"comment":"The claim that the first-stage retrieval result (A3, R@1 = 0.997) establishes a generation benefit is missing a link. Retrieval accuracy shows that the fusion embedding is well aligned in the CLIP embedding space, but it does not by itself prove that injecting that embedding into the second-stage diffusion model improves synthesis. The second-stage ablation is the relevant test, and as noted above its gains are small and unreplicated. The authors should either provide a stronger second-stage ablation or explicitly frame the retrieval result as evidence of alignment rather than as evidence of generation quality.","section":"Fusion Embedding Conditioned Diffusion Model (Method)"}],"minor_comments":[{"comment":"The sentence 'In the first stage In the first stage, the CLIP image encoder is used...' contains a duplicated phrase that should be removed.","section":"Method"},{"comment":"The paragraph describing PCDM, RePoseDM, and X-MDPT repeats the same sentence twice ('PCDM ... use Transformer to perform PGPIS by deeply learning the relationship between the source and the target pose images').","section":"Related Work"},{"comment":"Table 2 is captioned 'Qualitative results of first-stage ablation' but reports quantitative Recall@k and average rank; the caption should say 'Quantitative results.'","section":"Ablation Study"},{"comment":"Equation (1) writes 'ϵ N(0, 1)', missing the distribution symbol; it should be 'ϵ ~ N(0, 1)'.","section":"Preliminary (Equation 1)"},{"comment":"The definition of FIDt is given only as 'measures the FID of the test dataset'; please specify precisely which feature distributions are compared and how the train/test shift is handled, so that the metric can be reproduced by other groups.","section":"Experiments (Metrics)"},{"comment":"The figure is referenced inconsistently as 'Figure 1(a)' and 'Figure 1 (B)' in the text; the labels should be unified.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The DeepFashion state-of-the-art claim is mostly supported by the reported numbers, and the code release is a positive feature. The main risk is the gap between the paper's strong novelty language ('substantially improves') and the small, unreplicated ablation deltas in Table 3. If the authors cannot supply error bars or a stronger ablation, they should soften the claim; the contribution would then be a useful engineering combination rather than a demonstrated new principle. The Phoenix hand-metric mismatch should be resolved before publication. The RGB-pose-representation issue is a correctness-risk concern that a short additional ablation can address within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FPDM is a sensible two-stage pipeline for pose-guided person image synthesis: a CLIP-based image-pose fusion module is trained with contrastive loss to align a source+pose embedding with the target image embedding, and that fusion embedding is then injected into a Stable Diffusion UNet alongside DINOv2 source features and a CNN pose encoding. The specific combination and the Source-Enhanced Pose Fusion loss are new to PGPIS, and the first-stage retrieval result (R@1 = 0.997) is genuinely strong evidence that the fusion embedding captures target identity. The method is evaluated on two benchmarks with code released, and the DeepFashion numbers are competitive or better than recent diffusion baselines on LPIPS/SSIM/PSNR. That part is earned.\n\nThe soft spots are mostly about claims vs evidence. The abstract and conclusion say the ablation 'substantially improves' texture fidelity, but Table 3 shows B6 over B4/B5 with LPIPS deltas around 0.002, SSIM around 0.002, PSNR around 0.06, and FID actually worse for B6 than B4. No error bars, no seeds, no significance tests. The paper's own Limitations section says the differences are 'noticeable but not substantial.' So the load-bearing claim of the paper — that explicit fusion alignment gives a real generation gain — is not established, even though the method is plausible. This is the main thing a referee would need the authors to fix: either proper replication with error bars or a scaled-back claim.\n\nTwo more minor issues. The Phoenix Hand Pose comparison is not apples-to-apples because the authors used HRNetv2 while SignGAN used Ge et al. — that metric column is not comparable. And the claim that a three-channel RGB pose image suffices is asserted from visual inspection only; a quick comparison against multi-channel pose encoding would settle it. The author list also differs between the abstract and the full text, which is sloppy and should be cleaned up.\n\nBottom line: the paper is a solid incremental contribution, likely usable as a baseline, and the code release is a plus. But it overclaims, and the central ablation needs reinforcement. It deserves peer review — a serious editor should send it out, with a request for error bars and toned-down wording. It is not a reject on the merits; it is a revise.","headline":"Solid two-stage PGPIS pipeline with a real but incremental contrastive fusion idea; the headline ablation claim is unsupported by tiny, unreplicated deltas.","tokens_in":13611,"tokens_out":2685,"would_cite":true,"duration_ms":27301,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes FPDM, a two-stage diffusion model that explicitly aligns a fused source-pose embedding with the target image embedding via contrastive learning, then uses that embedding to condition image generation.","keywords":["pose-guided person image synthesis","diffusion model","contrastive learning","fusion embedding","source-enhanced pose fusion","latent diffusion","sign language generation","DeepFashion"],"falsifier":"Train the same FPDM pipeline twice, once with the three-channel RGB pose image and once with a multi-channel pose encoding that separates joints or joint connections, keeping all other settings identical; if the RGB variant does not match or beat the multi-channel variant on pose-alignment metrics such as hand-keypoint error or LPIPS on complex poses, the paper's central conditioning premise fails. A second check is retrieval accuracy of the first stage on a held-out pose set: the R@1 claim of 0.997 should be reproduced when pose images are drawn from unseen signers or unusual body configurations.","tokens_in":12568,"feed_emoji":"🧍","tokens_out":6146,"duration_ms":57243,"temperature":0.7,"pith_summary":"FPDM is a two-stage diffusion method for pose-guided person image synthesis whose central claim is that generation is best driven by an explicit fusion embedding of the source image and the target pose, learned to match the target image's embedding before the diffusion model sees it. The first stage trains an image-pose fusion module with a contrastive loss that also pulls the source embedding into the batch, so the learned embedding separates person identity from pose. The second stage feeds the frozen fusion embedding into a latent diffusion U-Net together with source-image features and pose features, and the paper reports state-of-the-art LPIPS, SSIM, and PSNR on DeepFashion plus large gains over prior sign-language generation methods on the PHOENIX-2014T benchmark. The claimed payoff is consistency: the same target pose yields stable imagery across different source images, and the same source yields stable style across different poses.","feed_headline":"Fusion embedding tops pose-guided image synthesis scores","feed_subtitle":"It beats prior methods on texture fidelity and stays stable when pose or source changes.","key_machinery":"The Image-Pose Fusion (IPF) module is the load-bearing mechanism: it uses a CLIP vision encoder to embed source image, target pose, and target image, combines source and pose embeddings with a Combiner module, and trains the result against the target embedding with InfoNCE contrastive loss. The paper's Source-Enhanced Pose Fusion variant doubles the batch by treating each source embedding as its own target, which forces the embedding space to separate persons by pose rather than by identity. The second stage then conditions a latent diffusion U-Net with three inputs: DINOv2 features of the source image acting as keys and values in transformer blocks, a four-layer convolutional pose encoder following the ControlNet design, and the fusion embedding added to the timestep embedding in ResNet blocks. Classifier-free guidance is applied with separate weights for the source-pose conditioning and the fusion embedding.","core_discovery":"The paper claims to be the first to explicitly align a fused source-pose embedding with the target image embedding via contrastive learning and then use that aligned embedding as a conditioning signal for pose-guided diffusion. On DeepFashion, its best model reaches LPIPS 0.1445, SSIM 0.7417, and PSNR 18.832 at 256x176, outperforming the compared state of the art on those semantic and pixel-level metrics, and on the PHOENIX sign-language set it reports SSIM 0.886 and FID 5.129 against 0.759 and 27.75 for the best prior method. The first-stage retrieval experiment, with R@1 of 0.997, is presented as evidence that the fusion embedding is genuinely aligned with the target. The paper also claims robustness: images stay consistent when the source image or the target pose varies.","pith_inferences":["Editorial inference: the contrastively aligned fusion embedding is a general conditioning idea, not tied to person images; the same two-stage alignment could be tested for object reposing, virtual try-on with different garment views, or any image geometry-to-appearance task.","Editorial inference: the paper's own tables show the B3-to-B6 differences are modest, so the headline gains on DeepFashion may owe more to scaling the source encoder input to 512 than to the fusion embedding itself; ablations that hold the source encoder fixed while toggling only the IPF loss would isolate this.","Editorial inference: the shared CLIP encoder for images and poses is a resource-driven choice; separate encoders fine-tuned for pose geometry would make the RGB-pose assumption more testable and could improve pose fidelity on extreme angles.","Editorial inference: a testable extension is using the learned fusion embedding for zero-shot or few-shot pose transfer across unseen identities, since the source-enhanced contrastive objective trains the embedding to be identity-independent."],"forward_implications":["The two-stage recipe yields state-of-the-art LPIPS, SSIM, and PSNR on DeepFashion at 256x176 and best LPIPS at 512x352, so texture fidelity and pixel alignment improve without waiting for implicit diffusion learning.","The same model transfers to sign-language frames, reporting SSIM 0.886 and FID 5.129 on PHOENIX-2014T, far ahead of prior video and generation baselines, which suggests the conditioning mechanism works where precise hand poses matter.","The stage-two-only model already approaches state-of-the-art performance, so the paper offers its structure as a simple, reusable baseline for future pose-guided person image synthesis diffusion models.","Because the fusion embedding is aligned to the target before generation, the model is robust to source-image and pose variations, as shown in the qualitative ablations.","The first-stage fusion retrieval reaches R@1 of 0.997, meaning the embedding alone nearly identifies the correct target image, directly supporting its use as a generation condition."],"supporting_citations":[{"why":"Supplies the Combiner module that fuses source-image and pose embeddings into a single fusion embedding.","marker":"(Baldrati et al., 2023)"},{"why":"Defines the InfoNCE contrastive loss used to align the fusion embedding with the target-image embedding.","marker":"(Oord et al., 2018)"},{"why":"Provides the latent diffusion model that the second-stage conditional generator is built on.","marker":"(Rombach et al., 2022)"},{"why":"Supplies the CLIP vision encoder used to embed source, pose, and target images in the first stage.","marker":"(Radford et al., 2021)"},{"why":"Supplies the four-layer convolutional pose-encoder design adapted for the target-pose condition.","marker":"(Zhang et al., 2023)"},{"why":"Supplies the DINOv2 source-image encoder whose features act as keys and values in the U-Net transformer blocks.","marker":"(Oquab et al., 2023)"},{"why":"The progressive conditional diffusion model whose structure FPDM simplifies for the second stage.","marker":"(Shen et al., 2024)"},{"why":"Provides the DeepFashion benchmark and its train/test split used for the main evaluation.","marker":"(Liu et al., 2016)"},{"why":"Provides the PHOENIX-2014T sign-language dataset used to test hand-pose fidelity.","marker":"(Camgoz et al., 2018)"}],"fun_headline_variants":["Fusion embedding improves pose-guided person synthesis","Contrastive fusion boosts pose-guided diffusion synthesis","Fusion embedding enhances texture fidelity in pose synthesis","Aligning fusion embeddings for pose-guided person images","Fusion embedding lifts pose-guided synthesis performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single three-channel RGB skeleton image of the target pose carries enough geometric detail for both the CLIP encoder and the four-layer CNN pose encoder to drive accurate pose-controlled generation, even though the paper does not quantitatively compare this representation against multi-channel per-joint pose encodings.","fun_headline_variants_meta":{"raw":{"variants":["Fusion embedding improves pose-guided person synthesis","Contrastive fusion boosts pose-guided diffusion synthesis","Fusion embedding enhances texture fidelity in pose synthesis","Aligning fusion embeddings for pose-guided person images","Fusion embedding lifts pose-guided synthesis performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000603,"raw_usage":{"total_tokens":2830,"prompt_tokens":973,"completion_tokens":1857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1788}},"tokens_in":589,"tokens_out":1857,"duration_ms":27051,"temperature":1.0,"reasoning_tokens":1788,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:23.115302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same FPDM pipeline twice, once with the three-channel RGB pose image and once with a multi-channel pose encoding that separates joints or joint connections, keeping all other settings identical; if the RGB variant does not match or beat the multi-channel variant on pose-alignment metrics such as hand-keypoint error or LPIPS on complex poses, the paper's central conditioning premise fails. A second check is retrieval accuracy of the first stage on a held-out pose set: the R@1 claim of 0.997 should be reproduced when pose images are drawn from unseen signers or unusual body configurations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Combiner module that fuses source-image and pose embeddings into a single fusion embedding."},{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; and Clark, J","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP vision encoder used to embed source, pose, and target images in the first stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The progressive conditional diffusion model whose structure FPDM simplifies for the second stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DeepFashion benchmark and its train/test split used for the main evaluation."},{"cited_title":"C.; Hadfield, S.; Koller, O.; Ney, H.; and Bowden, R","cited_arxiv_id":null,"evidence_quote":"Provides the PHOENIX-2014T sign-language dataset used to test hand-pose fidelity."}],"review_version":1}