REVIEW 4 major objections 6 minor 2 cited by
Towards Zero-Shot Differential Morphing Attack Detection with Multimodal Large Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Multimodal LLMs can perform differential morphing attack detection zero-shot, with ChatGPT-4o outperforming Gemini on real biometric data.
desk verdict First D-MAD benchmark with multimodal LLMs on real data; the qualitative failure analysis is worth reading, but the reported accuracy numbers rest on an unspecified fusion rule and should be treated as illustrative, not measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key machinery is a deliberately engineered Chain-of-Thought (CoT) prompt that assigns the LLM the role of an image forensic expert, guides it through five visual checks (facial geometry, artifacts, identity consistency, natural capture, hairlines and marks), and demands a forced binary answer to two questions plus a 0–100 probability score and an explanation. Around this prompt, the framework uses three independent inference attempts per image pair, averages the probability scores for distribution analysis, and applies a logical OR across attempts as the binary decision rule for computing MACER and BPCER. This combination converts a general-purpose multimodal LLM into a structured, explainable D-MAD decision-maker.
What would settle it
Compute calibration error between each model's 0–100 confidence scores and ground-truth morph labels on a larger labeled dataset, or compare the model's score ranking against a panel of forensic experts ranking the same image pairs by morphing likelihood; if the scores are not well-calibrated or do not rank-correlate with expert judgment, the claimed zero-shot detection performance cannot be trusted.
Extended reading notes
Core claim
The paper's central discovery is that a multimodal LLM can act as a zero-shot differential morphing attack detector when given a pair of facial images—a trusted capture and a possibly morphed reference—and a prompt that forces a binary decision, a 0–100 confidence score, and a step-by-step forensic rationale. On 150 bona fide–morphed pairs and 50 bona fide–bona fide pairs from 54 real subjects, ChatGPT-4o achieves half total error rates of 21.5% (landmark morphing), 3.5% (diffusion-based PIPE), and 0% (GAN-based MIPGAN2), while Gemini shows higher error rates and more overlap between bona fide and morph score distributions. The paper also finds that repeated inference with a logical OR decision rule and Chain-of-Thought prompting improves reliability, but that both models exhibit failures-to-answer, contradictory outputs, and hedging.
Load-bearing premise
The load-bearing premise is that the confidence scores the models give themselves reflect actual morphing likelihood; if these numbers are arbitrary or biased, the reported detection errors do not measure real detection performance.
Editorial extensions
If this is right
- If the claim holds, zero-shot LLM-based D-MAD can be deployed without training data, offering immediate detection baselines for newly invented morphing techniques.
- The explainable outputs could give border-control operators natural-language rationales for alarms, not just a score.
- The logical OR fusion across repeated attempts suggests a cheap inference-time strategy to reduce LLM failures and hallucination-driven misses.
- The strong ChatGPT-4o performance on GAN morphs indicates that some general-purpose vision-language models already encode cues that specialized detectors need to be trained to find.
- The observed vulnerability of both models to landmark-based morphs defines a concrete gap for future prompt or model improvements.
Reading between the lines
- A calibrated version of these LLM confidence scores could be plugged into existing D-MAD pipelines as an additional feature, rather than as a standalone decision.
- The failure-to-answer and contradiction patterns suggest that ensembling multiple LLMs or fusing with a classical detector would likely improve robustness more than further prompt tuning alone.
- Since the data is limited to 54 subjects and passport-style captures, the striking MIPGAN2 result should be re-tested on large public morph benchmarks before being taken as a general capability.
- The conflicting 'same identity yet morphed' outputs highlight that identity verification and morph detection are not consistently aligned in LLMs; a testable fix is to prompt for joint reasoning that explicitly conditions morph decisions on identity match.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents the first application of multimodal large language models (LLMs) to differential morphing attack detection (D-MAD) on real biometric data. ChatGPT-4o and Gemini are given two facial images with a Chain-of-Thought (CoT) prompt that asks for an identity-verification decision, a morph-detection decision, a 0-100 probability score, and a natural-language explanation. The evaluation uses 50 bona fide-bona fide pairs and 50 bona fide-morphed pairs for each of three morphing methods (LMA, MIPGAN2, PIPE), with three inference attempts per pair fused by logical OR. Tables I and II report MACER, BPCER, and HTER by morph type and model, and the qualitative section documents failure-to-answer cases and conflicting responses. The central claimed result is that ChatGPT-4o outperforms Gemini, especially on GAN-based morphs.
Significance. If the quantitative comparison were robust, this would be a useful and timely contribution: it demonstrates that a frozen multimodal LLM can perform zero-shot D-MAD while supplying human-readable explanations, which is a practical advantage over black-box detectors. The qualitative failure analysis (Tables III-V) is a genuine strength, as are the use of real subjects and the explicit reporting of failure modes. However, the evaluation protocol currently leaves the decision rule underspecified and the sample sizes are small, so the headline numbers in Tables I-II should be treated as preliminary. The significance of the paper therefore depends on the proposed revision.
major comments (4)
- [Section IV-A; Tables I-II] The decision rule behind Tables I-II is not fully specified. The text states that the binary decision uses a logical OR across three inference attempts, but it never states the threshold on the 0-100 probability score that turns one attempt into a flag, and it does not define how failure-to-answer responses are coded. This matters because ChatGPT-4o frequently fails to answer (Tables III and V) while Gemini rarely does. Under a discard-failures coding, the effective number of attempts differs per pair; under a non-flag coding, a high-failure model is penalized; under a flag coding, a single spurious response creates a false alarm. The OR rule can therefore inflate Gemini's BPCER while selectively rescuing ChatGPT-4o's MACER. Please specify the threshold and failure coding, and recompute Tables I-II under single-attempt, majority-vote, and OR policies with a fixed threshold. As written, the reported 0% HTER for ChatGPT-4o on MIPGAN2 is not a well-defined measurement.
- [Section IV-A] The sample size of 50 pairs per cell is too small to support the claimed differences without uncertainty quantification. A reported MACER of 0% for MIPGAN2 corresponds to 0 errors out of 50, whose 95% Clopper-Pearson interval extends to about 7%; the ChatGPT-4o versus Gemini difference (0% vs 13% MACER) is not formally established. Report exact binomial or bootstrap confidence intervals for MACER, BPCER, and HTER, and use a paired test if the same subjects contribute multiple pairs. Without intervals, the reader cannot distinguish a genuine model difference from sampling noise.
- [Section III] The zero-shot claim is weakened by the prompt-development procedure. The paper says the prompt was 'iteratively refined through trial and error,' but it does not state whether the same evaluation pairs were used during that refinement. If they were, the prompt has effectively been tuned on the test set, so the evaluation measures the performance of a hand-adapted prompt rather than a zero-shot system. Please either specify that a separate development set was used, or split the data and report results on an untouched held-out subset.
- [Section V-A; Tables I-II] The paper provides no comparison with existing D-MAD methods. Even if the LLM comparison is internally consistent, the claim that LLM-based D-MAD is 'promising' requires a reference point: for example, a handcrafted-feature/SVM baseline or a deep Siamese method from the cited literature, evaluated under the same protocol and threshold. Add at least one conventional baseline so that the absolute HTER values and the trade-off between MACER and BPCER can be interpreted.
minor comments (6)
- [Section I; Figure 1] The word 'Diffrential' in the Figure 1 caption and the word 'quantitaive' in Section VI-A are typos.
- [Section III] The prompt contains the spacing errors 'ALW AYS' in two places; these should read 'ALWAYS'.
- [Section I] The introduction's roadmap says Section II presents the framework and Section IV the database, but Section III (prompt design) is omitted from the roadmap.
- [Section V-B; Tables III-IV] The tables of round-wise responses do not identify which image pair or morph type each example came from, making the qualitative claims hard to verify. Add image identifiers or morph-type labels.
- [References] Reference [9] is malformed ('F. landmark based face morphing. Open CV.') and should be completed or replaced with a proper citation.
- [Overall] For reproducibility, state the exact model versions, API access dates, and decoding parameters (e.g., temperature, max tokens) for both ChatGPT-4o and Gemini; LLM responses are stochastic and these details materially affect the results.
Circularity Check
No significant circularity: this is an empirical benchmark against external pre-trained LLMs, with no fitted parameter or self-citation chain that forces the reported results.
full rationale
The paper contains no derivation chain in which a predicted quantity is defined in terms of its own outcome. The central results are measured outputs of two external, pre-trained multimodal LLMs (ChatGPT-4o and Gemini) on fixed image pairs, so the HTER/MACER/BPCER values are not constructed from the inputs by definition. Self-citations (e.g., [16], [30], [32], [33]) supply the morph-generation protocols, dataset construction guidance, and a prior S-MAD study; none of these is used to justify the D-MAD detection numbers themselves, and none is a uniqueness theorem that forbids alternatives. The iterative prompt refinement described in Section III is a test-set-contamination / zero-shot framing concern rather than a circular reduction, because the prompt is not a parameter fitted to the labels and then renamed a prediction. The logical-OR fusion rule and the unspecified probability threshold in Section IV-A are protocol ambiguities that affect interpretability and reproducibility, but they do not make the decision equal to the input score by construction. No step meets the required bar of quoting a specific equation or fitted parameter that reduces to the target result.
Assumptions & free parameters
free parameters (3)
- number_of_inference_attempts =
3
- binary_fusion_rule =
logical OR across attempts
- prompt_design =
hand-crafted CoT prompt, iteratively refined
assumptions (5)
- domain assumption LLM probability scores are treated as calibrated vulnerability scores
- domain assumption CoT prompting improves answer reliability and reasoning
- domain assumption The first image in each pair is a known bona fide image
- domain assumption LMA, MIPGAN2, and PIPE morphs are representative of realistic attacks
- domain assumption The collected smartphone images approximate passport enrollment conditions
Cite this review
Pith. "Pith review of Towards Zero-Shot Differential Morphing Attack Detection with Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/Z7FRZABK
@misc{pith2026250515332,
author = {Pith},
title = {Pith review of: Towards Zero-Shot Differential Morphing Attack Detection with Multimodal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z7FRZABK}},
note = {Machine review of arXiv:2505.15332}
}
read the original abstract
Leveraging the power of multimodal large language models (LLMs) offers a promising approach to enhancing the accuracy and interpretability of morphing attack detection (MAD), especially in real-world biometric applications. This work introduces the use of LLMs for differential morphing attack detection (D-MAD). To the best of our knowledge, this is the first study to employ multimodal LLMs to D-MAD using real biometric data. To effectively utilize these models, we design Chain-of-Thought (CoT)-based prompts to reduce failure-to-answer rates and enhance the reasoning behind decisions. Our contributions include: (1) the first application of multimodal LLMs for D-MAD using real data subjects, (2) CoT-based prompt engineering to improve response reliability and explainability, (3) comprehensive qualitative and quantitative benchmarking of LLM performance using data from 54 individuals captured in passport enrollment scenarios, and (4) comparative analysis of two multimodal LLMs: ChatGPT-4o and Gemini providing insights into their morphing attack detection accuracy and decision transparency. Experimental results show that ChatGPT-4o outperforms Gemini in detection accuracy, especially against GAN-based morphs, though both models struggle under challenging conditions. While Gemini offers more consistent explanations, ChatGPT-4o is more resilient but prone to a higher failure-to-answer rate.
Figures
Forward citations
Cited by 2 Pith papers
-
DifFoundMAD: Foundation Models meet Differential Morphing Attack Detection
DifFoundMAD improves differential morphing attack detection by replacing traditional embeddings with those from vision foundation models and applying class-balanced lightweight fine-tuning, cutting high-security error...
-
Benchmarking Foundation Models for Zero-Shot Biometric Tasks
A benchmark of 41 foundation models shows CLIP/OpenCLIP/BLIP2 embeddings reach near-90% zero-shot face verification and DINO reaches 97.55% on IITD-R iris without fine-tuning.
Reference graph
Works this paper leans on
-
[1]
P. Aghdaie, B. Chaudhary, S. Soleymani, J. Dawson, and N. Nasrabadi. Attention aware wavelet-based detection of morphed face images. In 2021 IEEE International Joint Conference on Biometrics (IJCB), pages 1–8, 2021
work page 2021
- [2]
-
[3]
E. Caldeira, G. Ozgur, T. Chettaoui, M. Ivanovska, P. Peer, F. Boutros, V . Struc, and N. Damer. Madation: Face morphing attack detection with foundation models, 2025
work page 2025
-
[4]
S. Clemens, S.Wojciech, A. Hilsmann, and P. Eisert. Accurate and robust neural networks for face morphing attack detection.Journal of Information Security and Applications, 53:102526, 2020
work page 2020
-
[5]
N. Damer, J. H. Grebe, S. Zienert, F. Kirchbuchner, and A. Kuijper. On the generalization of detecting face morphing attacks as anomalies: Novelty vs. outlier detection. In2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS), pages 1–5, 2019
work page 2019
-
[6]
M. Ferrara, A. Franco, and D. Maltoni. Face demorphing in the presence of facial appearance variations. InProc. of the 26th European Signal Processing Conf. (EUSIPCO). IEEE, September 2018
work page 2018
-
[7]
Information Technology - Biometric presentation attack detection - Part 3: Testing and Reporting
ISO/IEC JTC1 SC37 Biometrics.ISO/IEC 30107-3. Information Technology - Biometric presentation attack detection - Part 3: Testing and Reporting. International Organization for Standardization, 2017
work page 2017
-
[8]
C. Kraetzer, A. Makrushina, T. Neubert, M. Hildebrandt, and J. Dittmann. Modeling attacks on photo-ID documents and applying media forensics for the detection of facial morphing. InProc. Workshop on Information Hiding and Multimedia Security, pages 21– 32, 2017
work page 2017
Show all 34 references
-
[9]
landmark based face morphing
F. landmark based face morphing. Open CV.https://www.learnopencv.com/ face-morph-using-opencv-cpp-python/
-
[10]
FRVT morph web site.https://pages.nist.gov/ frvt/html/frvt_morph.html
NIST. FRVT morph web site.https://pages.nist.gov/ frvt/html/frvt_morph.html
-
[11]
Chatgpt.https://chat.openai.com/, 2024
OpenAI. Chatgpt.https://chat.openai.com/, 2024. [Ac- cessed: Jan. 11, 2024]
2024
-
[12]
Ortega-Delcampo, C
D. Ortega-Delcampo, C. Conde, D. Palacios-Alonso, and E. Cabello. Border control morphing attack detection with a convolutional neural network de-morphing approach.IEEE Access, 8:92301–92313, 2020
2020
-
[13]
Patwardhan, R
S. Patwardhan, R. Ramachandra, and S. Venkatesh. Empowering morphing attack detection using interpretable image-text foundation model. InComputer Vision and Image Processing, pages 163–169, Singapore, 2024. Springer Singapore
2024
-
[14]
Raghavendra, K
R. Raghavendra, K. Raja, and C. Busch. Detecting morphed face images. In2016 IEEE 8th Intl. Conf. on Biometrics: Theory, Applica- tions and Systems (BTAS). 8th IEEE Intl. Conf. on Biometrics: Theory, Applications and Systems (BTAS-2016), IEEE, September 2016
2016
-
[15]
Raghavendra, K
R. Raghavendra, K. Raja, and C. Busch. Algorithmic fairness in face morphing attack detection. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 410–418, 2022
2022
-
[16]
Raghavendra, K
R. Raghavendra, K. Raja, S. Venkatesh, and C. Busch. Face morphing versus face averaging: Vulnerability and detection. InProc. Intl. Joint Conf. on Biometrics (IJCB), 2017
2017
-
[17]
Raghavendra, K
R. Raghavendra, K. Raja, S. Venkatesh, and C. Busch. Transferable deep-CNN features for detecting digital and print-scanned morphed face images. InIEEE Conf. on Computer Vision and Pattern Recog- nition Workshops (CVPRW), pages 1822–1830, 2017
2017
-
[18]
Raghavendra, S
R. Raghavendra, S. Venkatesh, K. Raja, and C. Busch. Towards making morphing attack detection robust using hybrid scale-space colour texture features. InIEEE 5th Intl. Conf. on Identity, Security, and Behavior Analysis (ISBA). IEEE, January 2019
2019
-
[19]
Ramachandra and G
R. Ramachandra and G. Li. Multimodality for reliable single image based face morphing attack detection.IEEE Access, 10:82418–82433, 2022
2022
-
[20]
Ramachandra and G
R. Ramachandra and G. Li. Residual colour scale-space gradients for reference-based face morphing attack detection. In2022 25th International Conference on Information Fusion (FUSION), pages 1–
-
[21]
Ramachandra, S
R. Ramachandra, S. Venkatesh, G. Li, and K. Raja. Differential newborn face morphing attack detection using wavelet scatter network. In2023 5th International Conference on Bio-engineering for Smart Technologies (BioSMART), pages 1–4. IEEE, 2023
2023
-
[22]
Scherhag, A
U. Scherhag, A. Nautsch, C. Rathgeb, M. Gomez-Barrero, R. N. J. Veldhuis, L. Spreeuwers, M. Schils, D. Maltoni, P. Grother, S. Marcel, R. Breithaupt, R. Ramachandra, and C. Busch. Biometric systems under morphing attacks: Assessment of morphing techniques and vulnerability rep...
2017
-
[23]
Scherhag, C
U. Scherhag, C. Rathgeb, J. Merkle, and C. Busch. Deep face representations for differential morphing attack detection.IEEE Trans. on Information Forensics and Security, 2020
2020
-
[24]
J. M. Singh and R. Ramachandra. Fusion of deep features for differential face morphing attack detection at automatic border control gates. In2022 10th European Workshop on Visual Information Processing (EUVIP), pages 1–5. IEEE, 2022
2022
-
[25]
J. M. Singh and R. Ramachandra. Reliable face morphing attack detection in on-the-fly border control scenario with variation in image resolution and capture distance. In2022 IEEE International Joint Conference on Biometrics (IJCB), pages 1–10. IEEE, 2022
-
[26]
Soleymani, B
S. Soleymani, B. Chaudhary, A. Dabouei, J. Dawson, and N. Nasrabadi. Differential morphed face detection using deep siamese networks. InInternational Conference on Pattern Recognition, pages 560–572. Springer, 2021
2021
-
[27]
G. Team. Gemini: A family of highly capable multimodal models, 2024
2024
-
[28]
Venkatesh, R
S. Venkatesh, R. Ramachandra, K. Raja, and C. Busch. Face morphing attack generation and detection: A comprehensive survey.IEEE Transactions on Technology and Society, 2(3):128–145, Sept. 2021
2021
-
[29]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022
2022
-
[30]
Zhang, R
H. Zhang, R. Ramachandra, K. Raja, and C. Busch. Morph-pipe: Plugging in identity prior to enhance face morphing attack based on diffusion model. InNorsk IKT-konferanse for forskning og utdanning (NISK), volume 3, pages 1–6, 2023
2023
-
[31]
Zhang, R
H. Zhang, R. Ramachandra, K. Raja, and C. Busch. Generalized single-image-based morphing attack detection using deep represen- tations from vision transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1510– 1518, 2024
2024
-
[32]
Zhang, R
H. Zhang, R. Ramachandra, K. Raja, and C. Busch. Chatgpt encoun- ters morphing attack detection: Zero-shot mad with multi-modal large language models and general vision models, 2025
2025
-
[33]
Zhang, S
H. Zhang, S. Venkatesh, R. Raghavendra, K. Raja, N. Damer, and C. Busch. MIPGAN—Generating strong and high quality morphing attacks using identity prior driven GAN.IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3):365–383, 2021
2021
-
[34]
Zhang, X
N. Zhang, X. Liu, X. Li, and G.-J. Qi. Morphganformer: Transformer- based face morphing and de-morphing, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.