Pith. sign in

REVIEW 5 major objections 5 minor 74 references

CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CAD claims that jointly modeling lip-audio alignment and per-modality artifacts detects video deepfakes at 99.96% AUC on IDForge-v2 and 100% AUC on most unseen-manipulation splits of FakeAVCeleb.

desk verdict A practically strong multimodal deepfake detector with impressive benchmark numbers, but the information-theoretic derivation in Section 3.2 is invalid and needs to be corrected or removed. read the letter →

arxiv 2505.15233 v1 pith:JWNEWAZJ submitted 2025-05-21 cs.CV

Yuxuan Du , Zhendong Wang , Yuhao Luo , Caiyong Piao , Zhiyuan Yan , Hao Li , Li Yuan This is my paper · ORCID
classification cs.CV
keywords deepfakedetectionmultimodalcross-modalalignmentdistillationlip-speechsynchronizationaudio-visuallearningmutualinformationvideoforensics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that multimodal deepfakes leave two complementary kinds of evidence—modality-specific artifacts, such as facial blending boundaries or spectral distortions in synthetic audio, and modality-shared semantic misalignments, such as mismatches between lip movement and speech—and that detectors need both. It proposes CAD, a framework with a cross-modal alignment path that measures distributional mismatch between video and audio semantic features, and a cross-modal distillation path that lets each modality keep its own forensic details while borrowing from the other. The paper reports that CAD reaches 99.96% AUC on IDForge-v2 and 100% AUC on most cross-manipulation splits of FakeAVCeleb, outperforming unimodal and prior multimodal detectors, and that ablations confirm both paths contribute. If correct, the framework offers a general recipe: joint detection should maximize shared information and preserve modality-specific information rather than sacrifice one for the other.

What carries the argument

The central object is the dual-path CAD architecture. One path, cross-modal alignment, takes frozen CLIP visual and Whisper audio features, applies cross-attention to produce $x_{v2a}$ and $x_{a2v}$, and minimizes the KL divergence between their softmax distributions to learn lip-speech-level semantic synchronization. The other path, cross-modal distillation, applies SimSiam-style asymmetric projections to unimodal video and audio embeddings and minimizes a negative cosine-similarity distillation loss with L2 normalization, transferring audio knowledge to the video stream while preserving modality-specific traces. The mutual-information decomposition $I(x_1,y) = I(x_1,x_2,y) + H(x_1,y|x_2)$ is the stated justification: the alignment path targets the shared term, the distillation path targets the conditional term, and the two losses are summed with the classification loss.

What would settle it

Compute the three entropies that Lemma 2 equates—$H(x_1,x_2,y)$, $H(x_1,y)$, $H(x_2,y)$, and $H(y)$—on the CLIP/Whisper embeddings that CAD actually produces; if they differ by more than numerical error, equation (3) fails and the stated theoretical justification is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that a multimodal deepfake detector should simultaneously maximize the mutual information shared across modalities, $I(x_1,x_2,y)$, and the modality-specific conditional term $H(x_1,y|x_2)$, and that a single architecture can do both. CAD implements the shared term with frozen CLIP and Whisper encoders, cross-attention between audio and visual features, and a KL-divergence alignment loss that pulls the two semantic distributions together; it implements the specific term with a SimSiam-based distillation loss that transfers audio knowledge into the video stream while preventing the two representations from collapsing into one. On IDForge-v2, the paper reports 99.96% AUC and 99.63% AP; on FakeAVCeleb intra-manipulation, 99.0% accuracy and 99.6% AUC; and on cross-manipulation splits, 100% AUC on four of five categories and a 100% averaged AP/AUC, outperforming AVFF and RealForensics.

Load-bearing premise

The theoretical story assumes that the learned latent space literally contains the video and audio feature distributions, so that their combined entropy equals the entropy of the latent space alone; that containment is asserted without proof, and if it does not hold, the mutual-information equations that motivate the two losses do not follow.

Editorial extensions

If this is right

  • Multimodal deepfake detectors should treat lip-speech alignment and per-modality artifacts as complementary signals; dropping either path measurably hurts performance, per the paper's ablations.
  • On unseen manipulation types, the framework generalizes: it reports perfect or near-perfect scores on four of five held-out FakeAVCeleb forgery categories.
  • Video-only detectors can be strengthened by audio distillation at training time: the paper shows training on IDForge-v2 transfers to FaceShifter and Celeb-DF, which contain no audio channel.
  • Existing fusion strategies that only align shared semantics, or only concatenate specific features, leave a detectable gap that a joint objective closes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The architecture may work even if the information-theoretic derivation does not, since the two losses directly implement the two intuitions; the proof's Lemma 2 premise—that the learned latent space set-contains both modality distributions—is asserted without proof and is not generally true.
  • The same shared/specific decomposition could extend to other modality pairs, such as face-text or video-depth, wherever a pre-trained frozen encoder pair is available.
  • A testable prediction follows: on forgeries that are semantically well-synchronized but visually pristine, the distillation path should dominate, and on semantically misaligned but individually clean forgeries, the alignment path should dominate; per-sample attribution could be measured by the two losses.
  • The paper's own limitation note says current fusion and alignment capacity is constrained and proposes larger models and autoregressive language models; whether that scaling preserves or extends the 100% cross-manipulation scores is an open test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CAD, a multimodal video deepfake detection framework with two modules: a cross-modal alignment module (cross-attention plus KL divergence between CLIP and Whisper features) and a cross-modal distillation module (SimSiam loss), intended to capture both modality-shared semantic consistency and modality-specific forensic artifacts. The method is evaluated on FakeAVCeleb, IDForge-v2, and cross-manipulation splits, reporting state-of-the-art results such as 99.96% AUC on IDForge-v2 and 100% AUC on most FakeAVCeleb cross-manipulation settings. Section 3.2 provides an information-theoretic motivation based on mutual information decomposition, and Sections 3.3 and 3.4 connect the two losses to that decomposition.

Significance. If the reported numbers are reproducible, CAD is a strong and relatively simple multimodal detector that improves over prior audio-visual baselines by notable margins, and its modular decomposition into modality-shared and modality-specific cues is conceptually instructive. The use of frozen CLIP and Whisper encoders with LoRA adapters is computationally efficient and practically appealing. However, the paper's central theoretical claim is not valid as written, and the experimental protocol contains a favorable 'either modality' evaluation rule and lacks variance estimates, so the significance of the empirical contribution depends on corrections and additional evidence.

major comments (5)
  1. [Section 3.2, Lemma 2] The containment assumption 'x1 ⊂ y and x2 ⊂ y' is not a well-defined relation for random variables, and the entropy equalities in Eq. (3) do not follow from it. For a concrete counterexample, take x1 and x2 as independent Bernoulli(1/2) variables and set y = x1 XOR x2; then H(y) = 1 bit but H(x1, y) = 2 bits, so H(x1, y) ≠ H(y). Consequently Eq. (4) is not a valid consequence of the stated premises. Since Sections 3.3 and 3.4 motivate the dual-path architecture by the terms in Eq. (5), the theoretical foundation of the method is unsupported. The authors should either provide a rigorous derivation under explicit assumptions (e.g., y is the joint variable (x1, x2)) or remove the information-theoretic justification and present the design as a heuristic.
  2. [Section 3.2, Theorem 1 / Eq. (6)] The derivation of Eq. (6) is mathematically incorrect. The last step replaces -E_{p(s,V,A,a,v)}[log p(v|V,A)] with +E_{p(v)}[H(p(v|V))]; these are not equal in general, since H(p(v|V)) is an entropy of a conditional distribution, not the cross-entropy between the marginal p(v) and p(v|V,A). Together with the first term, the expression evaluates to something of the form -KL(p(v|V) || p(v|a,V,A)), which is not the mutual information I(v;a|V,A). The proof also silently assumes that v depends on V and s only and that a depends on A and s only. This invalidates the conclusion of Theorem 1 and the subsequent interpretation of the two terms as unimodal and joint objectives. Please correct the derivation or discard the theorem.
  3. [Section 4.2, evaluation protocol] The decision rule for audio-visual methods (a video is classified as fake if either modality is manipulated) is more permissive than the rule for unimodal visual methods (fake only when the visual modality is forged). On FakeAVCeleb, which contains real-video/fake-audio samples, this rule gives multimodal methods an advantage that is not attributable to their architecture. The paper should report results separately for visual-only fakes, audio-only fakes, and both-modality fakes, or use a common evaluation criterion, so that the claimed superiority over unimodal baselines is not an artifact of the labeling rule.
  4. [Tables 4 and 5] The ablation tables show that CAD (Ours) achieves 99.6 AUC on FaceShifter while the 'onlyvideo' baseline achieves 100.0 AUC, yet the text states that 'the performance of the model utilizing only modality-specific features is significantly inferior to that with dual modality inputs.' This claim is contradicted by the FaceShifter column. The authors should acknowledge that on this video-only dataset the audio branch slightly reduces accuracy and discuss why, and adjust the wording accordingly.
  5. [Section 4.2, Tables 1-3] None of the experimental tables report variance or the number of independent runs. With results at or near ceiling (99.96% AUC, 100% AUC on most splits), it is impossible to determine whether the reported improvements over baselines are statistically meaningful. Please provide mean ± standard deviation over at least three seeds, or a clear statement about the number of runs, and consider reporting confidence intervals for the key comparisons.
minor comments (5)
  1. [Section 3.4, first paragraph] The phrase 'we applies distillation' should be corrected to 'we apply distillation.'
  2. [Conclusion] The phrase 'trans-membrane state distillation' appears to be a typographical error; the paper does not define or use 'trans-membrane' anywhere, and the intended phrase is likely 'cross-modal state distillation.'
  3. [Section 4.1, Training details] The training details do not specify the optimizer, learning rate, batch size, number of epochs, or the relative weights of the KL, distillation, and classification losses; these are needed for reproducibility.
  4. [Throughout] The paper switches between 'IDForge' and 'IDForge-v2' in the text and tables; please standardize the terminology.
  5. [Figure 2] Figure 2 is referenced but not described in the text in enough detail to understand the data flow between the alignment and distillation modules; a more detailed caption or a short description in Section 3.1 would help.

Circularity Check

1 steps flagged · score 2.0 of 10

No significant circularity in the reported benchmarks; one theoretical lemma reduces to its own definition but is not load-bearing for the empirical results.

  1. self definitional [Section 3.2, Lemma 2 and Eqs. (3)-(4)]
    "We also introduce y, a latent space that encapsulates both x1 and x2, ensuring full compatibility with video and audio representations. ... Since we define a total space distribution y that integrates both modalities, ensuring seamless compatibility with their respective representations, it follows that x1 ⊂ y and x2 ⊂ y. Consequently, the following equation is established: H(x1, x2, y) = H(x1, y) = H(x2, y) = H(y)."

    y is introduced specifically as the space that contains both x1 and x2, so the asserted equality H(x1,x2,y)=H(y) is a restatement of that definition rather than an independent result. Substituting it into Lemma 1 reduces the claimed three-way quantity I(x1,x2,y) to the pairwise mutual information H(x1)+H(x2)-H(x1,x2), i.e., I(x1;x2). The theoretical derivation of the shared/specific decomposition is thus built into the definition of y by construction. This is a partial circularity in the paper's theoretical motivation, but it is not used to compute any test-set metric; the reported AUC/AP values come from standard train/test evaluation, so the central empirical claim is not circular.

full rationale

The paper's headline numbers (99.96% AUC on IDForge-v2, 100% AUC on most FakeAVCeleb cross-manipulation splits) are empirical outcomes from standard training/test splits with frozen CLIP/Whisper encoders and standard losses; nothing in Section 3.2 is fitted to those test sets or used to generate those numbers. The self-citation to [62] is limited to the spatiotemporal adapter design and is not used to import a uniqueness claim or to forbid alternative designs, so it is not load-bearing circularity. The one genuinely definitional move is Lemma 2: y is defined as the latent space that contains x1 and x2, so the entropy equality H(x1,x2,y)=H(y) restates that definition, and Eq. (4) collapses the three-variable mutual information to I(x1;x2). The derivation also has independent correctness problems (the containment is asserted without proof for a learned latent, and the proof of Eq. (6) yields a negative KL rather than I(v;a|V,A)), but these are validity concerns rather than evidence that the reported detection accuracy was manufactured from its inputs. The empirical SOTA claim remains an independent finding, so the overall circularity score is low.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim does not introduce new physical or conceptual entities. Its main hidden loads are the unproved information-theoretic containment in Lemma 2, the unproved decomposition in Lemma 3, and the evaluation rule that a video is fake when either modality is manipulated. Several hyperparameters (loss weights, LoRA values, frame count) are underreported.

free parameters (4)
  • KL loss weight
    The total loss is not defined; no weighting coefficients for L_KL and L_KD are reported, so the balance between alignment and distillation is an unstated hyperparameter that determines the results.
  • LoRA rank r = 8
    Set to 8 by default in Section 4.1 without ablation; changes audio encoder adaptation and affects performance.
  • LoRA alpha = 16
    Set to 16 by default; not ablated.
  • Number of frames = 16
    16 frames are used per sample to align with Whisper's 30-second audio context; a manual choice affecting the temporal window.
assumptions (4)
  • standard math Mutual information identity H(V)=H(V|A)+I(V,A)
    Invoked in Section 3.2 Eq. (1) without proof; standard but requires well-defined random variables for video and audio.
  • ad hoc to paper Total space distribution y contains both x1 and x2, implying H(x1,x2,y)=H(y)
    Lemma 2 in Section 3.2; asserted without proof and generally false for a learned latent embedding, load-bearing for the derivation of Eq. (4).
  • domain assumption A video is classified as fake if either modality, or both, is manipulated
    Section 4.2 defines the evaluation protocol for audio-visual baselines; this rule can inflate the apparent advantage of multimodal detectors.
  • domain assumption Frozen CLIP and Whisper embeddings capture lip-sync and artifact-relevant features
    Sections 3.3-3.4 assume these pretrained representations are suitable for forensic alignment and distillation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation." pith.science (2026). https://pith.science/paper/JWNEWAZJ

@misc{pith2026250515233,
  author       = {Pith},
  title        = {Pith review of: CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JWNEWAZJ}},
  note         = {Machine review of arXiv:2505.15233}
}
read the original abstract

The rapid emergence of multimodal deepfakes (visual and auditory content are manipulated in concert) undermines the reliability of existing detectors that rely solely on modality-specific artifacts or cross-modal inconsistencies. In this work, we first demonstrate that modality-specific forensic traces (e.g., face-swap artifacts or spectral distortions) and modality-shared semantic misalignments (e.g., lip-speech asynchrony) offer complementary evidence, and that neglecting either aspect limits detection performance. Existing approaches either naively fuse modality-specific features without reconciling their conflicting characteristics or focus predominantly on semantic misalignment at the expense of modality-specific fine-grained artifact cues. To address these shortcomings, we propose a general multimodal framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD). CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates feature conflicts during fusion while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio). Extensive experiments on both multimodal and unimodal (e.g., image-only/video-only)deepfake benchmarks demonstrate that CAD significantly outperforms previous methods, validating the necessity of harmonious integration of multimodal complementary information.

Figures

Figures reproduced from arXiv: 2505.15233 by the authors.

Figure 1
Figure 1. Venn diagram illustrating different types [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of CAD. Our proposed CAD is designed to maximize and fully mine both [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visual illustrations of our method. Left: The visualization results of modality-specific learning and modality-shared learning by CAM [71]. Origin denotes the original video input. Modality-specific shows the attention distribution within the vision unimodal encoder, where attention focuses mainly on visual artifacts. Instead, Modality-shared illustrates the attention distribution when both modalities are aligned, w… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 57 canonical work pages

  1. [1]

    Mesonet: a compact facial video forgery detection network

    Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: a compact facial video forgery detection network. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–7, 2018

    work page 2018

  2. [2]

    A review of modern audio deepfake detection methods: challenges and future directions

    Zaynab Almutairi and Hebah Elgibreen. A review of modern audio deepfake detection methods: challenges and future directions. Algorithms, 15(5):155, 2022

    work page 2022

  3. [3]

    Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization

    Zhixi Cai, Kalin Stefanov, Abhinav Dhall, and Munawar Hayat. Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization. In 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1–10, 2022

    work page 2022

  4. [4]

    Quo vadis, action recognition? a new model and the kinetics dataset

    João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017

    2017

  5. [5]

    Hong-Shuo Chen, Mozhdeh Rouhsedaghat, Hamza Ghani, Shuowen Hu, Suya You, and C.-C. Jay Kuo. Defakehop: A light-weight high-performance deepfake detector. In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2021

    work page 2021

  6. [6]

    Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection

    Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18689–18698, 2022

    work page 2022

  7. [7]

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020

    arXiv 2002

  8. [8]

    Exploring simple siamese representation learning

    Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15745– 15753, 2021

    work page 2021

Show all 74 references
  1. [9]

    Sophia Koepke, Ying Shan, and Zeynep Akata

    Yanbei Chen, Yongqin Xian, A. Sophia Koepke, Ying Shan, and Zeynep Akata. Distilling audio-visual knowledge by compositional contrastive learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7012–7021, 2021

    2021

  2. [10]

    V oice-face homogeneity tells deepfake

    Harry Cheng, Yangyang Guo, Tianyi Wang, Qi Li, Xiaojun Chang, and Liqiang Nie. V oice-face homogeneity tells deepfake. ACM Transactions on Multimedia Computing, Communications and Applications, 20:1 – 22, 2022

    2022

  3. [11]

    Can we leave deepfake data behind in training deepfake detector? arXiv preprint arXiv:2408.17052, 2024

    Jikang Cheng, Zhiyuan Yan, Ying Zhang, Yuhao Luo, Zhongyuan Wang, and Chen Li. Can we leave deepfake data behind in training deepfake detector? arXiv preprint arXiv:2408.17052, 2024

    2024 arXiv

  4. [12]

    Xception: Deep learning with depthwise separable convolutions

    François Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1800–1807, 2017. 10

    2017

  5. [13]

    Not made for each other- audio-visual dissonance-based deepfake detection and localization

    Komal Chugh, Parul Gupta, Abhinav Dhall, and Ramanathan Subramanian. Not made for each other- audio-visual dissonance-based deepfake detection and localization. In Pradeep K. Atrey and Zhu Li, editors, Proceedings of the 28th ACM International Conference on Multimedia, pages 4...

    2020

  6. [14]

    Audio-visual person-of-interest deepfake detection

    Davide Cozzolino, Alessandro Pianese, Matthias Nießner, and Luisa Verdoliva. Audio-visual person-of-interest deepfake detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 943–952, 2023

    2023

  7. [15]

    Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge

    S. Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. Implicit identity leakage: The stumbling block to improving deepfake detection generalization. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994–4004, 2022

    2023

  8. [16]

    www.github.com/MarekKowalski/FaceSwap Accessed 2021-04-24

    FaceSwap. www.github.com/MarekKowalski/FaceSwap Accessed 2021-04-24

    2021

  9. [17]

    Self-supervised video forensics by audio-visual anomaly detection

    Chao Feng, Ziyang Chen, and Andrew Owens. Self-supervised video forensics by audio-visual anomaly detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10491–10503, 2023

    2023

  10. [18]

    Joint 3d face reconstruction and dense alignment with position map regression network

    Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. ArXiv, abs/1803.07835, 2018

    2018 arXiv

  11. [19]

    Imagebind one embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180– 15190, 2023

    2023

  12. [20]

    Leveraging real talking faces via self-supervision for robust forgery detection

    Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Leveraging real talking faces via self-supervision for robust forgery detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14930–14942, 2022

    2022

  13. [21]

    Lips don’t lie: A generalisable and robust approach to face forgery detection

    Alexandros Haliassos, Konstantinos V ougioukas, Stavros Petridis, and Maja Pantic. Lips don’t lie: A generalisable and robust approach to face forgery detection. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5037–5047, 2021

    2021

  14. [22]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988, 2022

    2022

  15. [23]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022

    2022

  16. [24]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  17. [25]

    Avfakenet: A unified end-to-end dense swin transformer deep learning model for audio-visual deepfakes detection

    Hafsa Ilyas, Ali Javed, and Khalid Mahmood Malik. Avfakenet: A unified end-to-end dense swin transformer deep learning model for audio-visual deepfakes detection. Appl. Soft Comput., 136:110124, 2023

    2023

  18. [26]

    Information theory and statistical mechanics

    Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957

    1957

  19. [27]

    Hasam Khalid, Minhan Kim, Shahroz Tariq, and Simon S. Woo. Evaluation of an audio-video multimodal deepfake dataset using unimodal and multimodal detectors. Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection, 2021

    2021

  20. [28]

    Hasam Khalid, Shahroz Tariq, and Simon S. Woo. Fakeavceleb: A novel audio-video multi- modal deepfake dataset. ArXiv, abs/2108.05080, 2021. 11

    2021 arXiv

  21. [29]

    Fast face-swap using convolu- tional neural networks

    Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. Fast face-swap using convolu- tional neural networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 3697–3705, 2017

    2017

  22. [30]

    Face x-ray for more general face forgery detection

    Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

    2020

  23. [31]

    Audio anti-spoofing detection: A survey

    Menglu Li, Yasaman Ahmadiadli, and Xiao-Ping Zhang. Audio anti-spoofing detection: A survey. arXiv preprint arXiv:2404.13914, 2024

    2024 arXiv

  24. [32]

    Celeb-df: A large-scale challenging dataset for deepfake forensics

    Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A large-scale challenging dataset for deepfake forensics. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3204–3213, 2020

    2020

  25. [33]

    Lips are lying: Spotting the temporal inconsistency between audio and visual in lip-syncing deepfakes

    Weifeng Liu, Tianyi She, Jiawei Liu, Boheng Li, Dongyu Yao, Ziyou Liang, and Run Wang. Lips are lying: Spotting the temporal inconsistency between audio and visual in lip-syncing deepfakes. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  26. [34]

    Exploiting visual artifacts to expose deepfakes and face manipulations

    Falko Matern, Christian Riess, and Marc Stamminger. Exploiting visual artifacts to expose deepfakes and face manipulations. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pages 83–92, 2019

    2019

  27. [35]

    Emo- tions don’t lie: An audio-visual deepfake detection method using affective cues

    Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emo- tions don’t lie: An audio-visual deepfake detection method using affective cues. Proceedings of the 28th ACM International Conference on Multimedia, 2020

    2020

  28. [36]

    Does audio deepfake detection generalize? arXiv preprint arXiv:2203.16263, 2022

    Nicolas M Müller, Pavel Czempin, Franziska Dieckmann, Adam Froghyar, and Konstantin Böttinger. Does audio deepfake detection generalize? arXiv preprint arXiv:2203.16263, 2022

    2022

  29. [37]

    Nguyen, Junichi Yamagishi, and Isao Echizen

    Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Capsule-forensics: Using capsule networks to detect forged images and videos. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2307–2311, 2019

    2019

  30. [38]

    Towards universal fake image detectors that generalize across generative models

    Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24480–24489, 2023

    2023

  31. [39]

    Avff: Audio-visual feature fusion for video deepfake detection

    Trevine Oorloff, Surya Koppisetti, Nicolò Bonettini, Divyaraj Solanki, Ben Colman, Yaser Yacoob, Ali Shahriyari, and Gaurav Bharaj. Avff: Audio-visual feature fusion for video deepfake detection. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pa...

    2024

  32. [40]

    Gpt-4: A large multimodal model

    OpenAI. Gpt-4: A large multimodal model. https://openai.com/research/gpt-4, 2023. Accessed: 2023-01-21

    2023

  33. [41]

    Deepfake generation and detection: A benchmark and survey

    Gan Pei, Jiangning Zhang, Menghan Hu, Zhenyu Zhang, Chengjie Wang, Yunsheng Wu, Guangtao Zhai, Jian Yang, Chunhua Shen, and Dacheng Tao. Deepfake generation and detection: A benchmark and survey. arXiv preprint arXiv:2403.17881, 2024

    2024

  34. [42]

    Namboodiri, and C.V

    K R Prajwal, Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C.V . Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In MM ’20: Proceedings of the 28th ACM International Conference on Multimedia, pages 484–492, 2020

    2020

  35. [43]

    Audio-visual deep neural network for robust person verification

    Yanmin Qian, Zhengyang Chen, and Shuai Wang. Audio-visual deep neural network for robust person verification. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:1079–1092, 2021

    2021

  36. [44]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional...

    2021

  37. [45]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, 2022

    2022

  38. [46]

    Faceforensics++: Learning to detect manipulated facial images

    Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Niessner. Faceforensics++: Learning to detect manipulated facial images. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, 2019

    2019

  39. [47]

    A comprehensive overview of deepfake: Generation, detection, datasets, and opportunities

    Jia Wen Seow, Mei Kuan Lim, Raphaël CW Phan, and Joseph K Liu. A comprehensive overview of deepfake: Generation, detection, datasets, and opportunities. Neurocomputing, 513:351–371, 2022

    2022

  40. [48]

    Detecting deepfakes with self-blended images

    Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18720–18729, 2022

    2022

  41. [49]

    Representative forgery mining for fake face detection

    Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In CVPR, 2021

    2021

  42. [50]

    Exploring depth information for detecting manipulated face videos

    Haoyue Wang, Sheng Li, Ji He, Zhenxing Qian, Xinpeng Zhang, and Shaolin Fan. Exploring depth information for detecting manipulated face videos. CoRR, abs/2411.18572, 2024

    2024 arXiv

  43. [51]

    Tan, and Haizhou Li

    Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14653–14662, 2023

    2023

  44. [52]

    Deep spatial gradient and temporal depth learning for face anti-spoofing

    Zezheng Wang, Zitong Yu, Chenxu Zhao, Xiangyu Zhu, Yunxiao Qin, Qiusheng Zhou, Feng Zhou, and Zhen Lei. Deep spatial gradient and temporal depth learning for face anti-spoofing. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5041–5050, 2020

    2020

  45. [53]

    Altfreezing for more general video face forgery detection

    Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, and Houqiang Li. Altfreezing for more general video face forgery detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4129–4138, 2023

    2023

  46. [54]

    Deepfake video detection using convolutional vision transformer

    Deressa Wodajo and Solomon Atnafu. Deepfake video detection using convolutional vision transformer. ArXiv, abs/2102.11126, 2021

    2021 arXiv

  47. [55]

    Binaural audio-visual localization

    Xinyi Wu, Zhenyao Wu, Lili Ju, and Song Wang. Binaural audio-visual localization. Proceed- ings of the AAAI Conference on Artificial Intelligence, 35(4):2961–2968, May 2021

    2021

  48. [56]

    Identity- driven multimedia forgery detection via reference assistance

    Junhao Xu, Jingjing Chen, Xue Song, Feng Han, Haijun Shan, and Yu-Gang Jiang. Identity- driven multimedia forgery detection via reference assistance. In ACM Multimedia 2024, 2024

    2024

  49. [57]

    Tall: Thumbnail layout for deepfake video detection

    Yuting Xu, Jian Liang, Gengyun Jia, Ziming Yang, Yanhao Zhang, and Ran He. Tall: Thumbnail layout for deepfake video detection. In ICCV, pages 22658–22668, 2023

    2023

  50. [58]

    Transcending forgery specificity with latent space augmentation for generalizable deepfake detection

    Zhiyuan Yan, Yuhao Luo, Siwei Lyu, Qingshan Liu, and Baoyuan Wu. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8984–8994, 2024

    2024

  51. [59]

    Df40: Toward next-generation deepfake detection

    Zhiyuan Yan, Taiping Yao, Shen Chen, Yandan Zhao, Xinghe Fu, Junwei Zhu, Donghao Luo, Chengjie Wang, Shouhong Ding, Yunsheng Wu, et al. Df40: Toward next-generation deepfake detection. arXiv preprint arXiv:2406.13495, 2024

    2024 arXiv

  52. [60]

    Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation

    Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation. arXiv preprint arXiv:2504.02782, 2025

    2025 arXiv

  53. [61]

    Ucf: Uncovering common features for generalizable deepfake detection

    Zhiyuan Yan, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Ucf: Uncovering common features for generalizable deepfake detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22412–22423, 2023. 13

    2023

  54. [62]

    Generalizing deepfake video detection with plug-and-play: Video-level blending and spatiotemporal adapter tuning

    Zhiyuan Yan, Yandan Zhao, Shen Chen, Mingyi Guo, Xinghe Fu, Taiping Yao, Shouhong Ding, and Li Yuan. Generalizing deepfake video detection with plug-and-play: Video-level blending and spatiotemporal adapter tuning. arXiv preprint arXiv:2408.17065, 2024

    2024 arXiv

  55. [63]

    Avoid-df: Audio-visual joint learning for detecting deepfake

    Wenyuan Yang, Xiaoyu Zhou, Zhikai Chen, Bofei Guo, Zhongjie Ba, Zhihua Xia, Xiaochun Cao, and Kui Ren. Avoid-df: Audio-visual joint learning for detecting deepfake. IEEE Transactions on Information Forensics and Security, 18:2015–2029, 2023

    2015

  56. [64]

    Exposing deep fakes using inconsistent head poses

    Xin Yang, Yuezun Li, and Siwei Lyu. Exposing deep fakes using inconsistent head poses. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8261–8265, 2019

    2019

  57. [65]

    Audio deepfake detection: A survey

    Jiangyan Yi, Chenglong Wang, Jianhua Tao, Xiaohui Zhang, Chu Yuan Zhang, and Yan Zhao. Audio deepfake detection: A survey. arXiv preprint arXiv:2308.14970, 2023

    2023 arXiv

  58. [66]

    Learning natural consistency representation for face forgery video detection

    Daichi Zhang, Zihao Xiao, Shikun Li, Fanzhao Lin, Jianmin Li, and Shiming Ge. Learning natural consistency representation for face forgery video detection. In European Conference on Computer Vision, pages 407–424. Springer, 2024

    2024

  59. [67]

    Inclusion 2024 global multimedia deepfake detection: Towards multi-dimensional facial forgery detection

    Yi Zhang, Weize Gao, Changtao Miao, Man Luo, Jianshu Li, Wenzhong Deng, Zhe Li, Bingyu Hu, Weibin Yao, Wenbo Zhou, et al. Inclusion 2024 global multimedia deepfake detection: Towards multi-dimensional facial forgery detection. arXiv preprint arXiv:2412.20833, 2024

    2024 arXiv

  60. [68]

    Multi-attentional deepfake detection

    Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi-attentional deepfake detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2185–2194, 2021

    2021

  61. [69]

    Attention-based spatial-temporal multi-scale network for face anti-spoofing

    Wei Zheng, Mengyuan Yue, Shuhuan Zhao, and Shuaiqi Liu. Attention-based spatial-temporal multi-scale network for face anti-spoofing. IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3):296–307, 2021

    2021

  62. [70]

    Exploring temporal coherence for more general video face forgery detection

    Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. Exploring temporal coherence for more general video face forgery detection. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15024–15034, 2021

    2021

  63. [71]

    Learning deep features for discriminative localization

    Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, 2015

    2016

  64. [72]

    Makelttalk: speaker-aware talking-head animation

    Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions on Graphics (TOG), 39(6):1–15, 2023

    2023

  65. [73]

    Joint audio-visual deepfake detection

    Yipin Zhou and Ser-Nam Lim. Joint audio-visual deepfake detection. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14780–14789, 2021

    2021

  66. [74]

    Lan- guagebind: Extending video-language pretraining to n-modality by language-based semantic alignment

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, WANG HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Lan- guagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In The ...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.