{"id":"b383aa49-93b3-4cab-9848-2f2b69735ed6","arxiv_id":"2505.15332","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Multimodal LLMs can perform zero-shot differential morphing attack detection on real face data, with ChatGPT-4o outperforming Gemini, though accuracy is poor for landmark-based morphs.","lead":"Researchers tested whether off-the-shelf multimodal AI models like ChatGPT-4o and Gemini can spot digitally morphed passport photos by comparing them with live face images. ChatGPT-4o detected GAN-based morphs well but struggled with landmark-based morphs, and both models gave unreliable answers without tailored prompts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported HTERs hinge on an OR-over-three-attempts fusion rule with unstated failure handling; the 0% MIPGAN2 result may be an artifact of that protocol rather than genuine detection ability.","rationale":"The paper is a proof-of-concept benchmark: it compares two multimodal LLMs for zero-shot D-MAD on a small real dataset and reports both quantitative error rates and qualitative explanations. For the central claim to hold, the reported MACER/BPCER/HTER values must reflect genuine detection ability under a well-defined decision protocol. The weakest point is the evaluation protocol in Section IV-A. The OR-over-three-attempts rule is an unusual fusion choice, and the manuscript does not state how a probability score is thresholded into a binary attack decision, nor how refusal/proxy responses—which the paper itself documents as frequent for ChatGPT-4o—are treated. Because every quantitative result passes through this rule, the 0% HTER for MIPGAN2 and the gap between the two models could be artifacts of the fusion rather than of detection skill. This aligns only partially with the reader's weakest assumption: the reader focused on score calibration, but the more immediate problem is the unstated mapping from attempts to final binary labels, including failure handling. A second concern is that Section III reports the prompt was iteratively refined through trial and error, apparently without a separate validation set; this undermines the zero-shot generalization claim, though it does not by itself invalidate the model comparison. The small sample size (50 pairs per cell) and absence of confidence intervals further weaken the numeric comparisons. These issues are serious but not fatal: the qualitative findings and the basic observation that LLMs can provide reasoned morph decisions are still of interest. A conditional acceptance with a request to fix the protocol and rerun the numbers is the appropriate outcome, so the reader's verdict remains unchanged.","tokens_in":12244,"tokens_out":6577,"duration_ms":64535,"concrete_test":"Re-run the Section IV-A evaluation using only the first inference attempt per pair, with failed attempts counted explicitly as abstentions, and report MACER/BPCER/HTER at a fixed score threshold (e.g., probability > 50). Also recompute with majority voting over the three attempts. If ChatGPT-4o's MIPGAN2 HTER rises above 0% or the ChatGPT-vs-Gemini gap closes or reverses, the OR-fusion rule is responsible for the headline result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—ChatGPT-4o's 0% HTER on MIPGAN2 versus Gemini's 25.5%—is computed under the logical-OR fusion rule in Section IV-A: a pair is called an attack if any of three inference attempts flags it. This is not a standard D-MAD protocol, and the paper never specifies (i) the threshold on the 0–100 probability score that turns an attempt into a flag, or (ii) how failure-to-answer responses are coded in the OR. The paper reports that ChatGPT-4o frequently fails to answer (e.g., Table V: \"I'm unable to determine...\"), while Gemini rarely fails. If failed attempts are discarded, the effective number of attempts varies per pair and favors whichever model produces more usable outputs; if failures are coded as non-flags, models with high failure rates are penalized. With three attempts, the OR rule can selectively rescue ChatGPT-4o on morph pairs while the same rule inflates Gemini's BPCER because a single spurious flag is enough. The 50-pair-per-cell sample also yields no confidence intervals: a reported 0% error is compatible with a true error rate of several percent. The headline comparison is therefore not robust until the decision threshold, failure handling, and fusion rule are specified and metrics are recomputed under a single-attempt or majority-vote protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents the first application of multimodal large language models (LLMs) to differential morphing attack detection (D-MAD) on real biometric data. ChatGPT-4o and Gemini are given two facial images with a Chain-of-Thought (CoT) prompt that asks for an identity-verification decision, a morph-detection decision, a 0-100 probability score, and a natural-language explanation. The evaluation uses 50 bona fide-bona fide pairs and 50 bona fide-morphed pairs for each of three morphing methods (LMA, MIPGAN2, PIPE), with three inference attempts per pair fused by logical OR. Tables I and II report MACER, BPCER, and HTER by morph type and model, and the qualitative section documents failure-to-answer cases and conflicting responses. The central claimed result is that ChatGPT-4o outperforms Gemini, especially on GAN-based morphs.","tokens_in":12552,"tokens_out":7007,"duration_ms":61540,"significance":"If the quantitative comparison were robust, this would be a useful and timely contribution: it demonstrates that a frozen multimodal LLM can perform zero-shot D-MAD while supplying human-readable explanations, which is a practical advantage over black-box detectors. The qualitative failure analysis (Tables III-V) is a genuine strength, as are the use of real subjects and the explicit reporting of failure modes. However, the evaluation protocol currently leaves the decision rule underspecified and the sample sizes are small, so the headline numbers in Tables I-II should be treated as preliminary. The significance of the paper therefore depends on the proposed revision.","major_comments":[{"comment":"The decision rule behind Tables I-II is not fully specified. The text states that the binary decision uses a logical OR across three inference attempts, but it never states the threshold on the 0-100 probability score that turns one attempt into a flag, and it does not define how failure-to-answer responses are coded. This matters because ChatGPT-4o frequently fails to answer (Tables III and V) while Gemini rarely does. Under a discard-failures coding, the effective number of attempts differs per pair; under a non-flag coding, a high-failure model is penalized; under a flag coding, a single spurious response creates a false alarm. The OR rule can therefore inflate Gemini's BPCER while selectively rescuing ChatGPT-4o's MACER. Please specify the threshold and failure coding, and recompute Tables I-II under single-attempt, majority-vote, and OR policies with a fixed threshold. As written, the reported 0% HTER for ChatGPT-4o on MIPGAN2 is not a well-defined measurement.","section":"Section IV-A; Tables I-II"},{"comment":"The sample size of 50 pairs per cell is too small to support the claimed differences without uncertainty quantification. A reported MACER of 0% for MIPGAN2 corresponds to 0 errors out of 50, whose 95% Clopper-Pearson interval extends to about 7%; the ChatGPT-4o versus Gemini difference (0% vs 13% MACER) is not formally established. Report exact binomial or bootstrap confidence intervals for MACER, BPCER, and HTER, and use a paired test if the same subjects contribute multiple pairs. Without intervals, the reader cannot distinguish a genuine model difference from sampling noise.","section":"Section IV-A"},{"comment":"The zero-shot claim is weakened by the prompt-development procedure. The paper says the prompt was 'iteratively refined through trial and error,' but it does not state whether the same evaluation pairs were used during that refinement. If they were, the prompt has effectively been tuned on the test set, so the evaluation measures the performance of a hand-adapted prompt rather than a zero-shot system. Please either specify that a separate development set was used, or split the data and report results on an untouched held-out subset.","section":"Section III"},{"comment":"The paper provides no comparison with existing D-MAD methods. Even if the LLM comparison is internally consistent, the claim that LLM-based D-MAD is 'promising' requires a reference point: for example, a handcrafted-feature/SVM baseline or a deep Siamese method from the cited literature, evaluated under the same protocol and threshold. Add at least one conventional baseline so that the absolute HTER values and the trade-off between MACER and BPCER can be interpreted.","section":"Section V-A; Tables I-II"}],"minor_comments":[{"comment":"The word 'Diffrential' in the Figure 1 caption and the word 'quantitaive' in Section VI-A are typos.","section":"Section I; Figure 1"},{"comment":"The prompt contains the spacing errors 'ALW AYS' in two places; these should read 'ALWAYS'.","section":"Section III"},{"comment":"The introduction's roadmap says Section II presents the framework and Section IV the database, but Section III (prompt design) is omitted from the roadmap.","section":"Section I"},{"comment":"The tables of round-wise responses do not identify which image pair or morph type each example came from, making the qualitative claims hard to verify. Add image identifiers or morph-type labels.","section":"Section V-B; Tables III-IV"},{"comment":"Reference [9] is malformed ('F. landmark based face morphing. Open CV.') and should be completed or replaced with a proper citation.","section":"References"},{"comment":"For reproducibility, state the exact model versions, API access dates, and decoding parameters (e.g., temperature, max tokens) for both ChatGPT-4o and Gemini; LLM responses are stochastic and these details materially affect the results.","section":"Overall"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely within the scope of a biometrics or computer-vision venue, and the novelty claim (first LLM-based D-MAD on real data) is plausible. My main concern is that a revision could superficially add justifying text for the OR fusion rule without redoing the numbers. I would ask the editor to require the threshold/failure-handling specification and recomputed metrics under a conventional protocol before any acceptance decision. The prompt-tuning-on-test-data issue also deserves explicit treatment in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nIf you work on morphing attack detection, this is the first paper to point multimodal LLMs at differential MAD on real subject data. That alone makes it worth a skim. The more durable part is the qualitative analysis: they document refusal-to-answer behavior, contradictory identity-vs-morph outputs, disclaimers, and a Gemini failure that returned a Base64 image instead of an analysis. That is genuinely useful material for anyone trying to deploy LLMs in biometric pipelines.\n\nThe quantitative claims, though, do not hold up as reported. The headline numbers (especially ChatGPT-4o's 0% HTER on MIPGAN2) come from a logical OR over three inference attempts, with no threshold stated for what counts as a \"flag\" and no statement of how a failed response is coded. Table V shows ChatGPT-4o frequently fails to answer, while Gemini rarely does; the comparison is therefore sensitive to whether failures are discarded, counted as bona fide, or treated as flags. The 50-pair-per-cell sample also gives no confidence intervals: 0% is compatible with a true error of several percent. On top of that, the prompt was iteratively refined on the same evaluation data (Section III), so \"zero-shot\" is doing more work than it should, and there is no comparison against specialized D-MAD methods. These are not marginal quibbles; they affect every number in Tables I and II.\n\nThat said, the central qualitative claim—ChatGPT-4o separates scores better than Gemini—is plausibly visible in the KDE plots, so the paper is not empty. The authors also acknowledge several of these limitations in the discussion, which is to their credit.\n\nThis deserves a serious referee: it is a first-of-a-kind benchmark with useful qualitative observations. But it needs heavy revision before the quantitative results can be cited. I'd ask for single-attempt and majority-vote results, an explicit flag threshold and failure-coding rule, confidence intervals, and at least one specialized D-MAD baseline. And separate prompt design from evaluation.\n\nWorth a look before your next project meeting; I wouldn't design anything around the numbers.","headline":"First D-MAD benchmark with multimodal LLMs on real data; the qualitative failure analysis is worth reading, but the reported accuracy numbers rest on an unspecified fusion rule and should be treated as illustrative, not measured.","tokens_in":13065,"tokens_out":2538,"would_cite":true,"duration_ms":22016,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal LLMs can perform differential morphing attack detection zero-shot, with ChatGPT-4o outperforming Gemini on real biometric data.","keywords":["differential morphing attack detection","multimodal large language models","zero-shot detection","chain-of-thought prompting","ChatGPT-4o","Gemini","face morphing","explainable biometrics"],"falsifier":"Compute calibration error between each model's 0–100 confidence scores and ground-truth morph labels on a larger labeled dataset, or compare the model's score ranking against a panel of forensic experts ranking the same image pairs by morphing likelihood; if the scores are not well-calibrated or do not rank-correlate with expert judgment, the claimed zero-shot detection performance cannot be trusted.","tokens_in":12088,"feed_emoji":"🔎","tokens_out":4351,"duration_ms":34063,"temperature":0.7,"pith_summary":"The paper tries to establish that multimodal large language models, prompted as forensic experts to reason step-by-step, can perform differential morphing attack detection (D-MAD) in a zero-shot manner on real biometric data. It presents the first such application, comparing ChatGPT-4o and Gemini on passport-style image pairs from 54 consented subjects. The central claim is that these models show promising detection accuracy—particularly ChatGPT-4o against GAN-based morphs, with a 0% half total error rate for MIPGAN2—while also supplying natural-language explanations, although both models remain vulnerable under difficult conditions. The paper argues that Chain-of-Thought prompting reduces refusal rates and improves reasoning transparency, making LLM-based D-MAD a plausible complement to conventional detectors.","feed_headline":"ChatGPT-4o tops Gemini in zero-shot morph detection","feed_subtitle":"On 54 real subjects, Chain-of-Thought prompting lets a multimodal LLM flag morphed passport photos—and say why.","key_machinery":"The key machinery is a deliberately engineered Chain-of-Thought (CoT) prompt that assigns the LLM the role of an image forensic expert, guides it through five visual checks (facial geometry, artifacts, identity consistency, natural capture, hairlines and marks), and demands a forced binary answer to two questions plus a 0–100 probability score and an explanation. Around this prompt, the framework uses three independent inference attempts per image pair, averages the probability scores for distribution analysis, and applies a logical OR across attempts as the binary decision rule for computing MACER and BPCER. This combination converts a general-purpose multimodal LLM into a structured, explainable D-MAD decision-maker.","core_discovery":"The paper's central discovery is that a multimodal LLM can act as a zero-shot differential morphing attack detector when given a pair of facial images—a trusted capture and a possibly morphed reference—and a prompt that forces a binary decision, a 0–100 confidence score, and a step-by-step forensic rationale. On 150 bona fide–morphed pairs and 50 bona fide–bona fide pairs from 54 real subjects, ChatGPT-4o achieves half total error rates of 21.5% (landmark morphing), 3.5% (diffusion-based PIPE), and 0% (GAN-based MIPGAN2), while Gemini shows higher error rates and more overlap between bona fide and morph score distributions. The paper also finds that repeated inference with a logical OR decision rule and Chain-of-Thought prompting improves reliability, but that both models exhibit failures-to-answer, contradictory outputs, and hedging.","pith_inferences":["A calibrated version of these LLM confidence scores could be plugged into existing D-MAD pipelines as an additional feature, rather than as a standalone decision.","The failure-to-answer and contradiction patterns suggest that ensembling multiple LLMs or fusing with a classical detector would likely improve robustness more than further prompt tuning alone.","Since the data is limited to 54 subjects and passport-style captures, the striking MIPGAN2 result should be re-tested on large public morph benchmarks before being taken as a general capability.","The conflicting 'same identity yet morphed' outputs highlight that identity verification and morph detection are not consistently aligned in LLMs; a testable fix is to prompt for joint reasoning that explicitly conditions morph decisions on identity match."],"forward_implications":["If the claim holds, zero-shot LLM-based D-MAD can be deployed without training data, offering immediate detection baselines for newly invented morphing techniques.","The explainable outputs could give border-control operators natural-language rationales for alarms, not just a score.","The logical OR fusion across repeated attempts suggests a cheap inference-time strategy to reduce LLM failures and hallucination-driven misses.","The strong ChatGPT-4o performance on GAN morphs indicates that some general-purpose vision-language models already encode cues that specialized detectors need to be trained to find.","The observed vulnerability of both models to landmark-based morphs defines a concrete gap for future prompt or model improvements."],"supporting_citations":[{"why":"Supplies the ChatGPT-4o model that is one of the two multimodal LLMs under test.","marker":"[11]"},{"why":"Supplies the Gemini model that is the second multimodal LLM under test.","marker":"[27]"},{"why":"Provides the Chain-of-Thought prompting method that the paper's prompt design is based on.","marker":"[29]"},{"why":"Defines the MACER and BPCER metrics used for the quantitative D-MAD evaluation.","marker":"[7]"},{"why":"Specifies the protocol for subject pairing and constraints on morph generation used to build the dataset.","marker":"[16]"},{"why":"Describes the landmark-based morphing approach (LMA) used as one of the three attack types.","marker":"[9]"},{"why":"Introduces MIPGAN-2, the GAN-based morphing method used as the second attack type.","marker":"[33]"},{"why":"Presents the diffusion-based PIPE morphing approach used as the third attack type.","marker":"[30]"},{"why":"Earlier work applying ChatGPT-4o to single-image morphing attack detection, which this paper extends to the differential setting.","marker":"[32]"}],"fun_headline_variants":["ChatGPT-4o outguns Gemini in zero-shot morph detection","First zero-shot morph attack detection via multimodal LLMs","Chain-of-thought prompts sharpen LLM morph detection","LLM reveals morphed passports on real subjects","Multimodal LLM names morphs: ChatGPT-4o leads"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the confidence scores the models give themselves reflect actual morphing likelihood; if these numbers are arbitrary or biased, the reported detection errors do not measure real detection performance.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT-4o outguns Gemini in zero-shot morph detection","First zero-shot morph attack detection via multimodal LLMs","Chain-of-thought prompts sharpen LLM morph detection","LLM reveals morphed passports on real subjects","Multimodal LLM names morphs: ChatGPT-4o leads"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1245,"prompt_tokens":977,"completion_tokens":268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":593,"tokens_out":268,"duration_ms":3483,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:19:08.762707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute calibration error between each model's 0–100 confidence scores and ground-truth morph labels on a larger labeled dataset, or compare the model's score ranking against a panel of forensic experts ranking the same image pairs by morphing likelihood; if the scores are not well-calibrated or do not rank-correlate with expert judgment, the claimed zero-shot detection performance cannot be trusted.","supporting_citations":[{"cited_title":"Chatgpt.https://chat.openai.com/, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the ChatGPT-4o model that is one of the two multimodal LLMs under test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Gemini model that is the second multimodal LLM under test."},{"cited_title":"Information Technology - Biometric presentation attack detection - Part 3: Testing and Reporting","cited_arxiv_id":null,"evidence_quote":"Defines the MACER and BPCER metrics used for the quantitative D-MAD evaluation."},{"cited_title":"Raghavendra, K","cited_arxiv_id":null,"evidence_quote":"Specifies the protocol for subject pairing and constraints on morph generation used to build the dataset."},{"cited_title":"landmark based face morphing","cited_arxiv_id":null,"evidence_quote":"Describes the landmark-based morphing approach (LMA) used as one of the three attack types."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"Introduces MIPGAN-2, the GAN-based morphing method used as the second attack type."},{"cited_title":"Zhang, R","cited_arxiv_id":null,"evidence_quote":"Presents the diffusion-based PIPE morphing approach used as the third attack type."},{"cited_title":"Zhang, R","cited_arxiv_id":null,"evidence_quote":"Earlier work applying ChatGPT-4o to single-image morphing attack detection, which this paper extends to the differential setting."}],"review_version":1}