{"id":"442e3db9-354e-4482-b76d-c6b4efb1384f","arxiv_id":"2504.14202","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FaceCLIP-SDXL encodes identity and text into a single joint embedding and fully fine-tunes Stable Diffusion XL on it, reporting higher face similarity and text alignment than InstantID and PuLID-SDXL.","lead":"This paper introduces FaceCLIP, an encoder that combines a person's face identity and a text prompt into one representation that guides an image generation model. The authors report that their FaceCLIP-SDXL system produces portraits that keep the person's identity while following the text prompt better than two existing adapter-based methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Metric overlap with training objectives leaves the SOTA claim unproven: ArcFace and CLIP spaces are both alignment targets and evaluation metrics.","rationale":"The paper's core contribution is empirical: a new joint encoder and full fine-tuning recipe that claims to beat InstantID and PuLID-SDXL on identity preservation, text adherence, and image fidelity. The load-bearing condition is that the evaluation metrics measure the qualities claimed. Section 4.1 defines Face Similarity via ArcFace, and Section 3.2's Eq. (3) explicitly aligns the joint representation to ArcFace embeddings; CLIP-T uses the same kind of CLIP space that FaceCLIP is trained on, and possibly the exact same CLIP model, since the evaluation protocol does not say. Because FaceCLIP-SDXL is fully fine-tuned rather than adapter-injected, it has ample capacity to invert these alignment spaces: high cosine similarity in the trained embedding space does not guarantee that the generated face is perceptually the same person, and high CLIP-T does not guarantee the prompt's semantic content is present. The internal validation set adds an in-distribution risk, and the single user study (FaceCLIP vs PuLID only, one aggregate preference number) does not independently validate identity. These deficiencies are fixable and do not show the method is wrong; they do mean the SOTA statement is not yet fully supported. The proposed external, disjoint-embedding and human-judgment check would settle the concern. Thus no change to the reader's CONDITIONAL verdict is needed.","tokens_in":11489,"tokens_out":7760,"duration_ms":75887,"concrete_test":"Re-run the Table 2 comparison on an external, identity-disjoint public dataset (e.g., 50 identities from FFHQ or CelebA-HQ held out from training). Report Face Similarity with (a) ArcFace as in the paper, (b) a face-recognition embedding never used in any training pipeline, e.g., FaceNet or AdaFace, and (c) a forced-choice human identity-match judgment. Report CLIP-T with a CLIP model that is not one of the FaceCLIP backbones (e.g., ViT-B/32 if FaceCLIP uses L/14 and bigG-14). If FaceCLIP-SDXL's margins over InstantID and PuLID-SDXL persist on the unseen embedding and human judgment, the metric-overlap objection is resolved; if the margins shrink or invert, the SOTA claim is largely an artifact of aligned evaluation spaces.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The state-of-the-art claim rests on metrics that are partly constructed by the method itself. Face Similarity is cosine similarity in ArcFace space, and Eq. (3) explicitly trains FaceCLIP to align its joint representation to ArcFace embeddings via Lc(ec→r, ercls); the generative model is then fully fine-tuned on 43M samples to produce images from that representation. A model trained this way can learn to output faces whose ArcFace codes match the reference even when perceptual identity is weak, because ArcFace is an identity-discriminative but lossy embedding that collapses pose, lighting, and expression. The CLIP-T metric is also suspect: the paper never specifies which CLIP model computes it, and two candidate models (OpenCLIP-L/14 and bigG-14) are the same backbones used inside FaceCLIP; if either is used, the generation model is conditioned on the exact embedding space being scored. Additionally, the headline Face Similarity numbers come from Internal-v1, a validation set drawn from the same internal portrait data used for training, with no stated identity-disjointness check; Unsplash-50 is external but retains the same metric-overlap problem. The user study only compares FaceCLIP-SDXL against PuLID-SDXL and reports a single aggregate preference (68.6%), so it does not independently validate identity fidelity. Thus the observed margins over InstantID and PuLID-SDXL may be an artifact of optimizing the exact spaces in which the method is scored, rather than a generalizable improvement in identity preservation and text adherence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FaceCLIP, a multi-modal encoder that fuses an ArcFace-derived identity embedding with CLIP text and image embeddings into a joint representation, and FaceCLIP-SDXL, a pipeline that fully fine-tunes SDXL to condition on this representation. The pre-training objective (Eq. 3) aligns the joint representation with CLIP image, ArcFace identity, and CLIP text embedding spaces, and the authors add a guided image-text dataset to preserve text alignment. Experiments include zero-shot ImageNet classification, t-SNE visualization of identity clusters, ablations over pre-training losses, and comparisons with InstantID and PuLID-SDXL on Internal-v1 and Unsplash-50, reporting Face Similarity, CLIP-T, FID, and a user study. The paper claims state-of-the-art identity preservation, text adherence, and image fidelity.","tokens_in":11772,"tokens_out":4977,"duration_ms":42811,"significance":"The core idea of replacing adapter-based identity injection with a jointly trained identity-text conditioning encoder and fully fine-tuning the base diffusion model is a meaningful alternative to existing approaches, and the large-scale 43M-sample training plus guided-data regularization is an interesting engineering contribution. The paper also provides favorable evidence for the method's internal consistency: the ablation in Table 3 shows that adding identity alignment sharply changes face similarity, the t-SNE plots show identity structure after pre-training, and the guided dataset measurably improves zero-shot classification. However, the headline state-of-the-art claim is currently weakened by a systematic overlap between the spaces used in training objectives and the spaces used in evaluation metrics, which I detail in the major comments. If the authors can validate the results with independent metrics and identity-disjoint evaluation, the contribution would be solid and potentially significant for practical ID-preserving generation.","major_comments":[{"comment":"The evaluation metrics overlap with the training objectives in a load-bearing way. Face Similarity is the cosine similarity between ArcFace embeddings of reference and generated images, and Eq. (3) explicitly trains FaceCLIP to align its joint representation with the ArcFace embedding er_cls through Lc(ec->r, er_cls). Similarly, CLIP-T is computed as cosine similarity in a CLIP embedding space, while Eq. (3) also aligns ec->t with the CLIP text embedding et_cls and with the CLIP image embedding eI. Because the diffusion model is conditioned on this jointly trained e, the large Face Similarity margins over InstantID and PuLID-SDXL in Table 2 may reflect the model inverting the exact embedding spaces on which it was trained, rather than a generalizable improvement in perceived identity fidelity or text adherence. I am not claiming the method is invalid, but I do claim that the state-of-the-art conclusion is unproven without evaluation in spaces not used during training. Please report Face Similarity with a different face recognition backbone (e.g., AdaFace or FaceNet), compute CLIP-T with a frozen CLIP model that is not used anywhere in FaceCLIP or FaceCLIP-SDXL, and add a human identity-match and text-adherence test with per-criterion breakdowns.","section":"Section 4.1 (Evaluation) and Eq. (3)"},{"comment":"The Internal-v1 validation set does not appear to be identity-disjoint from the training data. The SPSV dataset is constructed from nine public datasets plus internal datasets, and Internal-v1 is described as an internal validation set with 15 identities and 200 prompts per identity. Since FaceCLIP-SDXL is fully fine-tuned on 43M samples from the same data collection, the high Face Similarity score of 0.869 on Internal-v1 may be partly due to memorization of identities seen during training. The paper states no identity-disjointness check between the training set and Internal-v1. Please report results on identities that are provably absent from the training data, or provide an external identity-disjoint benchmark, and report per-identity variance rather than only the aggregate Face Similarity.","section":"Section 4.1 (Datasets and Internal-v1)"},{"comment":"The paper does not specify which CLIP model is used to compute CLIP-T. This matters because Section 3.6 states that FaceCLIP-L/14 and FaceCLIP-bigG-14 use OpenAI-CLIP-L-14 and OpenCLIP-bigG-14 backbones, respectively; if CLIP-T is computed with either of these models, the metric evaluates the generated image in exactly the embedding space on which the conditioning encoder was trained. The current wording only says 'CLIP Score is computed as the cosine similarity between the generated image's embedding and the corresponding text prompt's embedding' without naming the model. Please specify the exact CLIP checkpoint and confirm that it is not used as a text or image encoder in FaceCLIP or FaceCLIP-SDXL.","section":"Section 4.1 (Evaluation, CLIP Score)"}],"minor_comments":[{"comment":"The sentence 'The diffusion model is then optimized according to Equation 3' appears to be a typo; the diffusion objective is defined in Eq. (2), while Eq. (3) is the pre-training loss for FaceCLIP.","section":"Section 3.6"},{"comment":"The text states that 'FaceCLIP encoders trained with guided data exhibit a notable performance drop,' but Table 1 shows that FaceCLIP-L-14 with guided data achieves 75.3 top-1 accuracy, slightly higher than OpenCLIP-L-14, and both guided variants dramatically outperform their no-guided counterparts (45.8 and 47.2). This sentence appears to refer to a comparison against the original OpenCLIP model for bigG only, or it is a typo; please clarify which comparison is meant.","section":"Section 4.2 and Table 1"},{"comment":"In the ablation table, L2 and L3 report identical Face Sim scores of 0.869; please report more decimal places or explain why adding the text alignment loss does not change identity similarity.","section":"Table 3"},{"comment":"The user study compares FaceCLIP-SDXL only with PuLID-SDXL, uses 17 participants, reports a single aggregate preference (68.6% vs. 27.8%), and provides no confidence intervals or per-criterion breakdown for identity similarity, text alignment, and image quality. This does not independently validate the identity-preservation claim.","section":"Section 4.4.1 (User Study)"},{"comment":"The guided-data mixing probability lambda is mentioned in Section 3.5 but never given a value, and several architecture details are missing, including the number of Feature Fusion blocks, the hidden dimension d, and the projection dimensions. Please add these implementation details or refer to released code.","section":"Section 3.5 and Section 4.1"},{"comment":"The FID values of 86.9 and 87.8 are high relative to typical FID ranges, and the paper does not state how many generated and reference images were used. Please specify the exact evaluation protocol for FID, including the number of samples.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong industrial-scale setup and a plausible method, and the authors should be encouraged to address the metric-overlap concern rather than treat it as a stylistic issue. The central SOTA claim rests on Face Similarity and CLIP-T scores that are computed in spaces used as training targets in Eq. (3); without independent metrics or an identity-disjoint benchmark, the reported margins cannot be taken at face value. I also note that the comparison set is narrow (only InstantID and PuLID-SDXL for the main claim), and the user study is limited. If the authors provide the requested independent evaluations, I would be willing to reconsider the paper favorably."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a serious methods paper with a real new idea, and its ablations are more convincing than the headline comparison. The authors replace adapter injection with a joint ID-text encoder, FaceCLIP, that fuses ArcFace identity embeddings with CLIP text embeddings via dual cross-attention, then fully fine-tunes SDXL on the unified condition. That is a genuine departure from InstantID/PuLID, and the ablation in Table 3 shows the ID alignment loss is doing real work: without it, Face Sim collapses to 0.077; with it, 0.869. The paper is well-written, builds cleanly on Arc2Face and CLIP, and the design could generalize beyond faces.\n\nWhere it gets soft: the evaluation metrics overlap with training objectives. Face Sim is cosine similarity in ArcFace space, and Eq. (3) explicitly trains the joint representation to align with ArcFace embeddings. CLIP-T is measured in CLIP space, and the CLIP backbones used for the score are the same ones used as the text encoders inside FaceCLIP. The paper never says which CLIP model computes CLIP-T, which is an omission that matters. Both headline numbers come from Internal-v1, an internal validation set drawn from the same portrait data used for training, with no identity-disjointness check. The user study is too narrow: it compares only FaceCLIP-SDXL vs PuLID-SDXL, reports a single aggregate preference, and doesn't independently validate identity fidelity. So the SOTA claim is real, but it's not proven by the evidence in the paper. The margins over InstantID and PuLID might partly reflect the generator inverting the exact embedding spaces being scored.\n\nThat said, I don't think this is a fatal circularity. ArcFace embeddings are intentionally lossy for identity; a model that optimizes ArcFace similarity can still produce perceptually poor faces. But the ablations and user study point in the same direction, and the method is plausible. The fix is straightforward: evaluate with a different face recognition model (e.g., a different ArcFace variant), report human judgments per criterion, add external validation sets with more identities, and specify the CLIP model for CLIP-T. I would also want a release of code or at least a demo, given the claims.\n\nWho this is for: anyone working on personalized T2I or multi-modal conditioning. It deserves serious peer review, and I'd send it out. I'd recommend major revision, not rejection, because the core method is sound and the deficiencies are fixable.","headline":"A genuinely new joint ID-text conditioning method with strong internal ablations, but the SOTA claim is undercut by evaluation metrics that overlap with training objectives.","tokens_in":12279,"tokens_out":2752,"would_cite":true,"duration_ms":23678,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing identity and text into a single joint embedding and fully fine-tuning SDXL on it lifts ID-preserving generation from about 0.66 to 0.87 face similarity over adapter baselines.","keywords":["identity-preserving image synthesis","joint ID-text representation","multimodal alignment","FaceCLIP","diffusion models","Stable Diffusion XL","contrastive learning","face identity embedding"],"falsifier":"Compute face similarity with a recognition backbone different from the one used in pre-training, for example a ViT-based face recognizer instead of ArcFace, and run a forced-choice human study on identities and prompts held out from training; if the reported 0.869 advantage shrinks toward the 0.65 baselines on either test, the central claim fails.","tokens_in":11305,"feed_emoji":"🖼️","tokens_out":6587,"duration_ms":52598,"temperature":0.7,"pith_summary":"This paper claims that identity-preserving portrait generation is better framed as sampling from a forward distribution conditioned on a single unified representation of both the person's identity and the text prompt, rather than as reconstruction through adapters bolted onto a frozen base model. To that end it introduces FaceCLIP, an encoder that fuses face patch features and text patch features into one joint embedding, and a pre-training loss that aligns this embedding to three spaces: CLIP image embeddings, CLIP text embeddings, and ArcFace identity embeddings. The authors then replace SDXL's two text encoders with trained FaceCLIP variants and fine-tune the full diffusion model on this joint condition. They report state-of-the-art face similarity (0.869 on Internal-v1, 0.866 on Unsplash-50), better CLIP scores, lower FID, and higher user preference than InstantID and PuLID-SDXL, with ablation evidence that identity preservation collapses when the identity alignment loss is removed.","feed_headline":"Fusing face identity and prompt text into one embedding beats adapters","feed_subtitle":"Fully fine-tuning SDXL on the joint representation raises face similarity from 0.66 to 0.87 over adapter methods.","key_machinery":"The central object is the FaceCLIP encoder $H_\\theta$, which turns a reference face crop and a text prompt into a fused sequence embedding $e \\in \\mathbb{R}^{b \\times L \\times d_c}$ via cascaded Feature Fusion blocks built on dual cross-attention plus self-attention. The load-bearing identity is the pre-training objective $L = L_c(e_{c\\to t}, e_I) + L_c(e_{c\\to r}, e_{r}^{cls}) + L_c(e_{c\\to t}, e_t^{cls})$: three contrastive losses that project the fused embedding and align it simultaneously with CLIP image embeddings, ArcFace identity embeddings, and CLIP text embeddings. This alignment is what makes the joint representation usable as a diffusion condition; the paper's ablation attributes the jump from 0.077 to 0.869 face similarity specifically to the ArcFace alignment term, and the text alignment term to the CLIP score gain (0.316 to 0.332). The second pillar is replacing both of SDXL's text encoders with FaceCLIP variants and fine-tuning the entire SDXL backbone, which distinguishes the method from adapter approaches that keep the foundation model fixed.","core_discovery":"The central claim is that identity and text should not be injected separately or via adapters; instead, a single joint representation $e = H_\\theta(c_t, c_r)$, produced by fusing identity and text patch embeddings through dual cross-attention, can carry both what the person looks like and what the prompt asks for. Trained with a sum of three contrastive losses—text-image, text-text, and face-identity alignment—this embedding remains discriminative for identity and still transfers well to text-only tasks such as zero-shot ImageNet classification. When SDXL is fully fine-tuned to condition on this joint representation, the paper argues, the model can generate photorealistic portraits that preserve identity and follow prompts more accurately than adapter-injection methods, because the whole generator adapts to the joint condition instead of a small plugin fighting a frozen backbone. The reported numbers support this: face similarity improves from about 0.66 to 0.87, and the ablation shows that removing the identity alignment loss drops face similarity to 0.077.","pith_inferences":["Editorial inference: the strongest untested implication is that the method's advantage may be partly metric-circular, since the pre-training aligns the joint embedding to ArcFace, the same space used to compute Face Similarity; a fair test would use an independent face-recognition backbone or human identity judgments on identities held out from training.","Editorial inference: the large reported scale, 43 million training samples and more than a hundred GPUs, suggests the gains may depend on compute and dataset size as much as on the architecture; a smaller-scale reproduction with the same losses on a public dataset would separate the algorithm's contribution from scale.","Editorial inference: the method should extend naturally to other subject types, such as objects or animals, by swapping the face encoder for a generic object encoder and aligning to a corresponding identity space, since nothing in the fusion or conditioning machinery is face-specific."],"forward_implications":["If the central claim holds, ID-preserving generation no longer needs adapters or test-time tuning: a single generator fine-tuned on a joint embedding can match or beat plugin-based pipelines, simplifying deployment.","The same joint-encoding recipe should transfer to other diffusion backbones by swapping the text encoders for FaceCLIP variants and fine-tuning, since the conditioning is just a fused embedding.","The reported ablation implies the identity alignment loss is the necessary ingredient: without it face similarity collapses to 0.077, so any reproduction must keep $L_c(e_{c\\to r}, e_r^{cls})$ in the pre-training objective.","Because the joint representation still retains text-only capability, with zero-shot ImageNet accuracy close to the OpenCLIP baselines, the trained encoder could serve as a drop-in replacement for the text encoder in ordinary text-to-image generation as well as ID-preserving synthesis."],"supporting_citations":[{"why":"Supplies the conditional-sampling formulation: Arc2Face trains a diffusion model on face embeddings as special tokens; the paper extends this from headshots to full portraits with text.","marker":"[18]"},{"why":"Supplies the contrastive alignment method and the pre-training objective pattern; the paper's $L_c$ is the CLIP contrastive loss.","marker":"[42]"},{"why":"Supplies the ArcFace identity embedding space used for $e_r^{cls}$ and for the Face Similarity metric.","marker":"[41]"},{"why":"Supplies the base diffusion model SDXL, whose two text encoders are replaced by FaceCLIP variants.","marker":"[17]"},{"why":"PuLID-SDXL is the main adapter-based baseline for comparison and the source of the Unsplash-50 benchmark.","marker":"[11]"},{"why":"InstantID is the other adapter-based baseline the paper must beat on identity and text metrics.","marker":"[12]"},{"why":"Supplies the caption annotations for the 43 million training tuples; removing it would break the training data construction.","marker":"[55]"}],"fun_headline_variants":["One joint embedding for face and text beats adapter injection","Fusing ID and prompt into a single condition raises ID fidelity","Joint condition outperforms adapters: ID similarity up from 0.66 to 0.87","SDXL fine-tuned on unified ID-text embedding preserves identity better","FaceCLIP: joint ID-text encoding beats adapters for portrait generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation metrics—ArcFace-based face similarity, CLIP score, and FID—actually measure genuine identity preservation and text adherence; if the high scores come mostly from the generator inverting the same embedding spaces the pre-training aligned to, the state-of-the-art claim would not survive a perceptual or independent test.","fun_headline_variants_meta":{"raw":{"variants":["One joint embedding for face and text beats adapter injection","Fusing ID and prompt into a single condition raises ID fidelity","Joint condition outperforms adapters: ID similarity up from 0.66 to 0.87","SDXL fine-tuned on unified ID-text embedding preserves identity better","FaceCLIP: joint ID-text encoding beats adapters for portrait generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00031,"raw_usage":{"total_tokens":1764,"prompt_tokens":937,"completion_tokens":827,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":732}},"tokens_in":553,"tokens_out":827,"duration_ms":7676,"temperature":1.0,"reasoning_tokens":732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:54:31.135081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute face similarity with a recognition backbone different from the one used in pre-training, for example a ViT-based face recognizer instead of ArcFace, and run a forced-choice human study on identities and prompts held out from training; if the reported 0.869 advantage shrinks toward the 0.65 baselines on either test, the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the conditional-sampling formulation: Arc2Face trains a diffusion model on face embeddings as special tokens; the paper extends this from headshots to full portraits with text."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive alignment method and the pre-training objective pattern; the paper's $L_c$ is the CLIP contrastive loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ArcFace identity embedding space used for $e_r^{cls}$ and for the Face Similarity metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PuLID-SDXL is the main adapter-based baseline for comparison and the source of the Unsplash-50 benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the caption annotations for the 43 million training tuples; removing it would break the training data construction."}],"review_version":1}