{"id":"1ada8580-a8bf-4ea6-b16b-50e329c5c7a4","arxiv_id":"2501.00700","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A frozen CLIP model with prompts guided by GPT-4 concepts plus test-time prompt tuning achieves 0.911 average AUC and 0.774 overall accuracy on the DeepFakeFaceForensics benchmark.","lead":"This paper proposes a method to detect AI-generated fake face images by teaching a vision-language model through prompts, using a language model to list telltale signs and then tuning those prompts on the test data itself. If the reported benchmark numbers hold, it offers a lightweight detector that adapts to new image types without retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transductive test-time prompt tuning makes the SOTA comparison protocol-asymmetric: the method sees each test subset before evaluation, while all baselines are inductive.","rationale":"The reader's conditional verdict is appropriate, and the weakest-assumption analysis aligns with mine. I examined the equations and tables for internal consistency; the method is clearly described, the 768-parameter count is consistent with N=1 and a 768-d CLIP token embedding, and the ablations at least show monotone behavior for KGP. The load-bearing weakness is not an arithmetic error but a missing control: the test-time prompt tuning consumes the test set, yet the headline comparisons are against inductive baselines. The ablation numbers strengthen this: without TTP, the method's OA (0.598) is below UnivFD (0.677), so the reported OA superiority is almost entirely created by adapting to the test data. This does not make the method useless—test-time adaptation can be legitimate—but it means the statement 'notably outperforms state-of-the-art methods' has not been demonstrated under a matched protocol. Adding a transductive baseline and oracle-label control would settle the point. I do not see grounds to reject; the concern is addressable and the paper is otherwise coherent.","tokens_in":8092,"tokens_out":6917,"duration_ms":70101,"concrete_test":"Add the identical TTP procedure (Eqs. 9-13, Treal=0.999, Tfake=0.5, TopK=128) to UnivFD by tuning its 769 trainable parameters on the same unlabeled test subsets, then compare full-method vs UnivFD+TTP. If UnivFD+TTP matches or surpasses the proposed AUC/OA, the claimed advantage is due to the transductive protocol, not KGP. As a second control, run TTP using ground-truth labels on exactly the same selected samples; if oracle-labeled TTP is not materially better than pseudo-label TTP, the pseudo-labels are not providing new information about the target domain.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('notably outperforms state-of-the-art methods', Section III-C) depends on a protocol asymmetry. In Section II-B, Eqs. (9)-(13), TTP uses the unlabeled test set: it selects high-confidence samples (Treal=0.999, Tfake=0.5, TopK=128) and updates prompts before AUC/OA are computed on the same subset. Every baseline in Table I, including UnivFD in Table III, is inductive and never touches test data. TTP is thus transduction, so the reported +0.098 AUC over GLFF and +0.220 OA over UnivFD conflates the proposed modules with extra test-set access. Table III makes the issue concrete: without TTP, OA falls from 0.774 to 0.598, below UnivFD's 0.677, while AUC falls only from 0.911 to 0.903. The large OA gain is therefore mostly test-time recalibration of the 0.5 threshold on target data, not evidence that the knowledge-guided prompt is better. Because pseudo-labels are generated by the same model and Tfake is only 0.5, the tuning loop can reinforce its own predictions (confirmation bias); the improvement may not reflect a genuine domain-shift correction. To support the SOTA claim, the authors must either compare against a transductive/test-time-adaptation baseline under the same protocol or report the inductive version as the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deepfake facial image detection method built on a frozen CLIP backbone with only 768 trainable parameters. It combines knowledge-guided prompt learning (KGP), which uses concepts retrieved from a large language model (e.g., GPT-4) to construct forgery-related prompts, with test-time prompt tuning (TTP), which selects high-confidence unlabeled test samples and tunes the prompts using pseudo-labels generated by the model itself. On the DeepFakeFaceForensics dataset, the method reports an average AUC of 0.911 and an overall accuracy of 0.774, outperforming several inductive baselines including GLFF and UnivFD. The central claim is that this represents a notable improvement over state-of-the-art methods for universal deepfake detection.","tokens_in":8353,"tokens_out":4278,"duration_ms":40293,"significance":"If the reported results hold under a fair protocol, the paper would make a useful contribution: it demonstrates that external knowledge from LLMs can be injected into prompt learning for deepfake detection with very few trainable parameters, and it provides a clean ablation showing that the knowledge-guided prompts alone (without test-time adaptation) already improve over a fixed-CLIP baseline. The idea of using LLM-retrieved semantic concepts to guide prompts is novel in this application and could transfer to other forensic tasks. However, the headline superiority claim is currently confounded by the transductive nature of the test-time tuning stage, which uses the unlabeled test set before evaluation while all baselines are inductive. The significance of the work therefore hinges on whether the evaluation protocol can be made fair and whether the gains from TTP can be attributed to genuine domain-shift correction rather than self-training artifacts.","major_comments":[{"comment":"The comparison with state-of-the-art methods is not apples-to-apples because the proposed method is transductive while every baseline in Tables I and II is inductive. Test-time prompt tuning (TTP) uses the unlabeled test set to select high-confidence samples via Eqs. (9)-(11), generates pseudo-labels, and updates the prompts with the loss in Eq. (13) before the reported AUC/OA are computed. All baselines never touch the test data. This protocol asymmetry is load-bearing: the ablation in Table III shows that removing TTP reduces average OA from 0.774 to 0.598, which is below UnivFD's 0.677, while AUC drops from 0.911 to 0.903. Thus the claimed superiority (e.g., +0.098 AUC over GLFF) is largely attributable to test-set access, not to the knowledge-guided prompt learning alone. The authors should either compare against test-time adaptation or transductive methods under the same protocol, or report the inductive variant as the headline result and treat the TTP gain as a separate contribution.","section":"Section II-B, Eqs. (9)-(13); Table III"},{"comment":"The test-time tuning loop is self-referential: the same model that generates pseudo-labels is then tuned on those labels. With Tfake = 0.5 and TopK = 128, the selected 'fake' set consists of any sample the model currently considers more likely fake than real, so the optimization can reinforce the model's own biases rather than correcting genuine domain shift. The paper does not analyze the accuracy of the pseudo-labels or compare against an oracle variant that uses ground-truth labels during test-time tuning. Without such an analysis, it is unclear whether the TTP improvement reflects a real adaptation to the target domain or merely confirmation bias. Please provide pseudo-label accuracy statistics and an oracle-label experiment to support the claim that TTP alleviates domain shift.","section":"Section II-B, Eqs. (9)-(10)"},{"comment":"It is not specified whether the reported AUC and OA are computed on the entire test subset or only on the selected high-confidence samples used for TTP. The selection procedure in Eqs. (9)-(11) with TopK=128 suggests that only a small subset may be used for tuning, but the evaluation could be on all test data in each 'Testing Subset' of Table I. If the metrics are restricted to the selected samples, the results are not comparable to baselines, which are evaluated on the full test distribution. The paper should explicitly state the evaluation set for each metric and, if necessary, report results on the full test set to make the comparison fair.","section":"Section II-B and Section III-A"}],"minor_comments":[{"comment":"The sentence 'we calculate the similarity scores and sort them in descending order' is ambiguous: it is unclear whether the sorting is done globally over all test samples or per class, and whether the selection is performed independently for each post-processing subset or over the entire test set. This affects the interpretation of TopK and the reproducibility of the method.","section":"Section II-B"},{"comment":"The notation for prompts is slightly confusing because Preal and P_fake^i both use [P1]...[PN], but the number of learnable prompts is later stated as 1 in the experiments. Please clarify how the shared learnable prompt tokens interact with the multiple LLM-retrieved concepts when N=1, and whether N denotes the number of tokens or the number of prompt vectors.","section":"Section II-A, Eqs. (1)-(2)"},{"comment":"There is a typo in the text: 'Comared' should be 'Compared' in the paragraph discussing trainable parameters. Additionally, the hyperparameter plots in Fig. 3 would benefit from error bars or repeated-run statistics, since the OA differences between some settings (e.g., 0.774 vs 0.764) are small and may not be significant.","section":"Section III-D"},{"comment":"The comparison set is somewhat narrow: beyond UnivFD, recent CLIP-based deepfake detectors such as FatFormer and AntiFakePrompt are mentioned in related work but not included in the quantitative comparison. Adding at least one recent prompt-based baseline would strengthen the claim of outperforming state-of-the-art methods.","section":"Section III-C"},{"comment":"The paper does not provide details of the LLM query, the exact set of retrieved concepts, or any filtering/validation process. For reproducibility, please include the query template and the list of concepts used for the fake prompt, or state that all retrieved concepts are used without filtering.","section":"Section II-A"}],"recommendation":"major_revision","confidential_remarks":"The protocol issue is the central concern: the method's reported superiority is largely driven by transductive test-time tuning, which is not matched by any baseline. However, the manuscript is honest in reporting the ablation that exposes this issue (Table III), and the knowledge-guided prompt learning component appears sound and well-motivated. I believe the paper can be revised to a publishable state by either (a) framing the inductive result as the main contribution and treating TTP as a separate transductive extension, or (b) adding transductive/test-time-adaptation baselines and analyzing pseudo-label quality. I would not recommend rejection because the core idea is novel and the experimental setup, except for the protocol asymmetry, is reasonable. The paper may also benefit from a more thorough comparison with recent prompt-based deepfake detectors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the LLM-elicited concept prompts are a nice addition to CLIP-based deepfake detection, and the ablations honestly show each module pulling its weight. But the headline 'notably outperforms' claim is built on a protocol asymmetry: the test-time tuning stage uses the unlabeled test set, while every baseline is inductive. That makes the numbers in Table I and the OA gain in Table III hard to trust as reported.\n\nWhat's genuinely new: combining LLM-retrieved forgery concepts with learnable prompts, and coupling that with test-time prompt tuning on a frozen CLIP. The method is simple and cheap: 768 trainable parameters, no backbone fine-tuning. The ablation separates KGP and TTP contributions: without KGP, AUC drops from 0.911 to 0.884, and without TTP, OA drops from 0.774 to 0.598. The hyperparameter study shows the method is not sensitive to thresholds or TopK within the tested ranges. That is all useful, reproducible-sounding work—though no code or exact LLM outputs are provided.\n\nThe soft spots are real and structural. The test-time prompt tuning selects high-confidence test samples by the model's own predictions (Eqs. 9–13) and then optimizes the same prompt used for prediction on those pseudo-labels. This is a self-training loop, and Tfake=0.5 is a loose threshold. More importantly, the comparison in every table is transductive for the proposed method and inductive for all baselines. Ablation makes the issue concrete: without TTP, OA falls to 0.598, below UnivFD's 0.677; AUC stays 0.903 vs UnivFD's 0.891. So the large OA advantage is mostly test-time recalibration toward the target data, not evidence that the knowledge-guided prompt is intrinsically better. The small AUC advantage without TTP is suggestive but not conclusive without error bars.\n\nTo make the SOTA claim stand, the authors should either report the inductive version as the headline result or compare against transductive/test-time adaptation baselines under the same protocol. A comparison with existing test-time prompt tuning methods would also help. These are addressable, so this is a conditional accept rather than a rejection.\n\nWho benefits: readers working on CLIP-based forgery detection, prompt learning, and test-time adaptation. It deserves serious referee time, but the requested revisions should be mandatory.\n\nMy recommendation: send it to review with a clear request to fix the evaluation protocol.","headline":"Clever idea, but the SOTA claim leans on a transductive test-time tuning stage that makes the comparison protocol-asymmetric.","tokens_in":8908,"tokens_out":2390,"would_cite":false,"duration_ms":21749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen CLIP with 768 trainable parameters detects deepfake faces at 0.911 average AUC, beating fully tuned state-of-the-art detectors on DeepFakeFaceForensics.","keywords":["deepfake detection","prompt learning","CLIP","test-time adaptation","large language model","domain shift","facial image forensics"],"falsifier":"Run the same test-time prompt tuning with the pseudo-labels randomly permuted among the same selected samples: if average AUC stays at 0.911 and overall accuracy at 0.774, the gain is not coming from learning real-versus-fake structure. Alternatively, give the comparison baselines the same transductive access to the unlabeled test batch (for example, entropy minimization or the same confident-sample selection without the LLM concepts) and check whether the 0.098 absolute AUC gap over GLFF persists; if it does, the protocol, not the knowledge guidance, carries the headline number.","tokens_in":7871,"feed_emoji":"🎭","tokens_out":8197,"duration_ms":67914,"temperature":0.7,"pith_summary":"This paper tries to establish that deepfake face detection improves when a frozen CLIP vision-language model is steered by two moves: prompts built from forgery-related concepts retrieved from a large language model, and a second round of prompt tuning on unlabeled test images using pseudo-labels the model assigns by its own confidence. The authors argue that prior detectors ignore both expert knowledge and the category-level domain shift between training images (natural objects such as horses and chairs) and test images (fine-grained human faces), and that these gaps can be closed without touching the backbone weights. If true, the result matters because it reports 0.911 average AUC and 0.774 overall accuracy on DeepFakeFaceForensics, an absolute AUC gain of 0.098 over the previous state of the art, using only 768 trainable parameters.","feed_headline":"768 parameters spot deepfake faces at 0.911 AUC","feed_subtitle":"LLM-steered prompts plus test-time tuning beat fully tuned detectors on the DeepFakeFaceForensics benchmark.","key_machinery":"The central objects are the two prompt templates $P_{real}$ and $P_{fake}^{i}$, each a learnable token sequence followed by a class description: $[real]$ for pristine images, or one of the LLM-retrieved concepts $\\phi_i$ from the set $\\Phi = \\{\\text{fake}, \\text{blurred}, \\dots, \\text{unrealistic}\\}$ for fakes. The frozen CLIP text encoder $G$ maps these to prototypes $G_{real}$ and $G_{fake} = \\frac{1}{|\\Phi|}\\sum_{i} G(P_{fake}^{i})$, while the frozen image encoder $F$ yields cosine similarities $S_{real}$ and $S_{fake}$; a temperature-scaled softmax converts them into the probabilities $\\delta_{real}$ and $\\delta_{fake}$ used in both the training loss and the test-time selection rule (Eqs. 5-13). The machinery's job is to turn a frozen vision-language model into a detector whose only movable parts are the prompts, so that detection can be re-adapted at inference time to the unlabeled test distribution.","core_discovery":"The discovery, on the paper's own terms, is that a single learnable prompt per class, seeded with LLM-retrieved forgery concepts, plus a transductive prompt-tuning stage on pseudo-labeled test data, yields top results on deepfake facial image detection. The fake-class prompt is not the bare word 'fake': it is an averaged text prototype over concepts such as 'blurred', 'unnatural' and 'inconsistent' collected by querying GPT-4, which the authors treat as expert-level prior knowledge that also makes the prompts interpretable. The test-time stage selects up to 128 high-confidence samples per class with thresholds $T_{real}=0.999$ and $T_{fake}=0.5$, assigns pseudo-labels, and continues optimizing the same prompts; the ablation attributes a large overall-accuracy jump (from 0.598 without test-time tuning to 0.774 with it) to this step.","pith_inferences":["The same two-stage recipe (LLM-derived concept prompts plus transductive prompt tuning) could plausibly transfer to other domain-shifted detection problems, such as synthetic audio or synthetic text detection, because nothing in the mechanism is face-specific.","A fairness test the authors do not report: how much of the 0.098 AUC gain survives when the comparison baselines are also allowed transductive access to the unlabeled test batch, or when test-time tuning runs with shuffled pseudo-labels; without that test, part of the gain may be protocol rather than detector quality.","The threshold asymmetry (0.999 for real vs. 0.5 for fake) suggests the selected 'real' samples are trusted far more than the selected 'fake' samples, so a natural extension is to test whether the tuning gain is driven mostly by the high-confidence real subset.","One could test the domain-shift story directly: on a test set drawn from the same distribution as training, the test-time tuning gain should shrink toward zero, and a large residual gain there would point to a mechanism other than shift correction."],"forward_implications":["A frozen CLIP ViT-L/14 with 768 trainable prompt parameters would outperform fully fine-tuned detectors, cutting deployment cost by orders of magnitude (0.911 vs. 0.813 average AUC against GLFF's 26.8M trained parameters).","Test-time prompt tuning on self-selected, pseudo-labeled test images would be a practical way to adapt a detector to a target domain at inference time without ground-truth labels.","LLM-retrieved forensic concepts would transfer across unseen generative models, since the presented results cover GAN-based, transformer-based and diffusion-based generators.","The ablations imply the two components contribute differently: knowledge-guided prompts lift both AUC and overall accuracy, while test-time tuning is what rescues accuracy under the domain shift.","The method would degrade gracefully across hyper-parameter settings according to the reported sweep, making it usable without per-dataset tuning."],"supporting_citations":[{"why":"UnivFD is the CLIP-based universal fake-image detector this method must beat; its fixed-CLIP probe protocol (769 trainable parameters) sets the direct comparison baseline.","marker":"[16]"},{"why":"CLIP is the frozen backbone; its text and image encoders supply all features and the similarity space used by both training and test-time tuning.","marker":"[17]"},{"why":"GLFF is the state-of-the-art baseline (26.8M trained parameters) it claims to outperform; the paper follows its training-set conventions and reports a 0.098 absolute AUC improvement over it.","marker":"[14]"},{"why":"GPT-4 is the large language model from which the forgery-related concepts (blurred, unnatural, inconsistent) are retrieved.","marker":"[21]"},{"why":"PromptAD supplies the evidence that enriched prompts with expert-level concepts boost vision-language model performance, motivating the knowledge-guided prompt design.","marker":"[20]"},{"why":"CNN-aug defines the training-set convention (LSUN categories with ProGAN images) and serves as a spatial-domain baseline in the comparison tables.","marker":"[3]"}],"fun_headline_variants":["LLM-retrieved prompts with test-time tuning hit 0.911 AUC on deepfakes","768 learnable parameters spot deepfake faces at 0.911 AUC","Prompt tuning with LLM knowledge beats full detectors on fake faces","Tiny prompt model uses GPT-4 concepts to detect deepfakes at 0.911"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that tuning prompts on test images the model itself marks as high-confidence is a valid inference-time step, and that the gain over inductive baselines is a fair measure of detection quality; if the self-generated pseudo-labels simply agree with the model's own mistakes, the reported improvement could reflect self-confirmation rather than genuine domain-shift correction.","fun_headline_variants_meta":{"raw":{"variants":["LLM-retrieved prompts with test-time tuning hit 0.911 AUC on deepfakes","768 learnable parameters spot deepfake faces at 0.911 AUC","Prompt tuning with LLM knowledge beats full detectors on fake faces","Tiny prompt model uses GPT-4 concepts to detect deepfakes at 0.911"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1684,"prompt_tokens":894,"completion_tokens":790,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":703}},"tokens_in":510,"tokens_out":790,"duration_ms":7330,"temperature":1.0,"reasoning_tokens":703,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:44:12.998536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same test-time prompt tuning with the pseudo-labels randomly permuted among the same selected samples: if average AUC stays at 0.911 and overall accuracy at 0.774, the gain is not coming from learning real-versus-fake structure. Alternatively, give the comparison baselines the same transductive access to the unlabeled test batch (for example, entropy minimization or the same confident-sample selection without the LLM concepts) and check whether the 0.098 absolute AUC gap over GLFF persists; if it does, the protocol, not the knowledge guidance, carries the headline number.","supporting_citations":[{"cited_title":"Towards universal fake image detectors that generalize across generative models,","cited_arxiv_id":null,"evidence_quote":"UnivFD is the CLIP-based universal fake-image detector this method must beat; its fixed-CLIP probe protocol (769 trainable parameters) sets the direct comparison baseline."},{"cited_title":"Learning transferable visual models from natural language supervision,","cited_arxiv_id":null,"evidence_quote":"CLIP is the frozen backbone; its text and image encoders supply all features and the similarity space used by both training and test-time tuning."},{"cited_title":"Glff: Global and local feature fusion for ai-synthesized image detection,","cited_arxiv_id":null,"evidence_quote":"GLFF is the state-of-the-art baseline (26.8M trained parameters) it claims to outperform; the paper follows its training-set conventions and reports a 0.098 absolute AUC improvement over it."},{"cited_title":"Promptad: Learning prompts with only normal samples for few-shot anomaly detection,","cited_arxiv_id":null,"evidence_quote":"PromptAD supplies the evidence that enriched prompts with expert-level concepts boost vision-language model performance, motivating the knowledge-guided prompt design."},{"cited_title":"Cnn- generated images are surprisingly easy to spot... for now,","cited_arxiv_id":null,"evidence_quote":"CNN-aug defines the training-set convention (LSUN categories with ProGAN images) and serves as a spatial-domain baseline in the comparison tables."}],"review_version":1}