{"id":"7dc37908-38dc-428b-a322-1e6aa3cd26fe","arxiv_id":"2505.08294","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A deepfake detector that injects frozen facial-action-unit features into a frame-wise audio-visual attention model achieves state-of-the-art results on FakeAVCeleb and LAV-DF, with the largest gains in cross-dataset generalization.","lead":"FauForensics detects audio-visual deepfakes by combining facial action units (muscle-movement descriptors) with frame-level audio-visual alignment. It reports state-of-the-art accuracy on two benchmarks and better cross-dataset generalization than prior detectors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-dataset SOTA may be inflated by train/test identity overlap and unmatched baseline protocol: FakeAVCeleb and LAV-DF are both VoxCeleb2-derived, and Table III does not control shared identities or FGMDF's 4s-clip filter.","rationale":"The paper's headline contribution is generalizability across datasets, and Table III is the sole quantitative support for the 4.83% claim. That evidence depends on two implicit assumptions: the datasets are disjoint in identities/content, and every method was evaluated under the same protocol. Neither assumption is verified in the manuscript. Because both datasets are VoxCeleb2-derived, identity overlap is plausible rather than hypothetical, and the FGMDF 4-second-clip caveat shows that at least one baseline was not evaluated under identical conditions. The unresolved 90.52 vs 89.61 numeric mismatch in the same section adds further uncertainty.\n\nI do not treat the audio-alignment limitation as the most load-bearing issue. It is explicitly acknowledged as a boundary condition, and the benchmark videos are talking-head clips with largely synchronized audio; it affects deployment scope but not the validity of the within-benchmark comparison. The unspecified FAU fusion f_phi is a serious reproducibility gap, but it would not invalidate the reported numbers if code were released; the protocol/identity issue can invalidate the central comparison itself.\n\nThe ablation study (Table IV) and the phoneme-AU25 visualization give independent, if partial, support for the FAU mechanism, so the paper should not be rejected on that basis. A single controlled re-run of Table III with identity-disjoint splits and a uniform clip-length filter would settle whether the concern lands. This reinforces the reader's CONDITIONAL verdict rather than changing it.","tokens_in":16528,"tokens_out":10117,"duration_ms":100463,"concrete_test":"Re-run the cross-dataset protocol with an explicit identity-disjoint split: (1) collect the VoxCeleb2 identity labels for every FakeAVCeleb and LAV-DF video from the dataset metadata; (2) report the intersection of identities between the LAV-DF train split and the FakeAVCeleb test split, and between the FakeAVCeleb train split and the LAV-DF test split; (3) if any overlap exists, retrain and retest on the restricted disjoint subsets and recompute Table III. In the same run, apply FGMDF's 4-second-clip filter to all methods (or report both filtered and unfiltered results) and recompute the average AUC gap. If the 4.83% gap shrinks materially, or if the identity intersection is large, the cross-dataset SOTA claim does not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 4.83% average cross-dataset AUC gain over FGMDF in Table III. That comparison is only meaningful if the two evaluation protocols are matched and if the two datasets are genuinely identity-disjoint. FakeAVCeleb [54] and LAV-DF [53] are both constructed from VoxCeleb2 talking-head videos, but the manuscript never reports whether the FakeAV test split shares identities or raw clips with the LAV-DF train split, or vice versa. If it does, the model can memorize real identities/videos during training and retrieve them at cross-dataset test time, inflating the reported 83.77/95.44 AUCs with memorization rather than FAU-guided audio-visual coherence. A second protocol gap compounds this: Section IV-B notes that FGMDF operates on 4-second clips and excludes shorter videos, and the 'fair comparison' using FGMDF's test list is shown only for the within-database confusion matrix in Fig. 4; Table III does not state whether the same test-list and clip-length filter was applied to all methods. The text also contains an unresolved numeric mismatch (Section IV-C reports 90.52% for the binary average while Table III shows 89.61%). These issues do not prove the method is wrong, but they leave the headline SOTA/generalization claim unsupported until the protocol is pinned down.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FauForensics, an audio-visual deepfake detection framework that combines a frozen facial action unit (FAU) encoder pretrained on DISFA with a trainable video encoder, a query-shared multimodal transformer for implicit feature alignment, and a temporal attentional pooler that computes frame-wise intra- and inter-modality attention matrices. The final predictions are produced by modality-specific MLPs, with the multimodal score used at inference. The authors report within-database and cross-database experiments on FakeAVCeleb and LAV-DF, perturbation robustness studies, ablations of the core modules and encoders, t-SNE visualizations, and a phoneme-FAU case study. The paper claims state-of-the-art performance and a 4.83% average cross-dataset AUC improvement over existing methods.","tokens_in":16843,"tokens_out":3336,"duration_ms":35796,"significance":"If the empirical claims hold, the paper would make a useful contribution by showing that biologically grounded facial action units can serve as a transferable representation for audio-visual deepfake detection and that frame-wise temporal correlation modeling can outperform clip-level approaches. The paper has real strengths: the FAU encoder is pretrained on an external dataset and frozen, so the detector is not circularly fitting the motivating temporal-correlation observation; the ablation study isolates the contributions of the proposed modules; the perturbation robustness experiments cover several realistic distortions; and the limitations paragraph honestly identifies the reliance on aligned audio-visual streams. However, the central cross-dataset generalization claim is currently under-supported because the comparison protocol is not fully specified, there is an unresolved numeric inconsistency in the reported gain, and some load-bearing components of the architecture are left undefined.","major_comments":[{"comment":"The fusion operation f_phi in Eq. (1) and the normalization function f_norm in Eqs. (4)-(6) are never defined, and it is not stated whether the learnable scaling factors sigma_av, sigma_a, and sigma_v are scalars or per-channel vectors. Since the paper attributes the cross-dataset improvement specifically to FAU-enhanced frame-wise audio-visual similarity and temporal attentional pooling, these unspecified operations are load-bearing for the central claim and make the method impossible to reimplement or fully check as written.","section":"III-B and III-D, Eqs. (1), (4)-(6)"},{"comment":"The cross-dataset comparison does not control for the identity overlap between FakeAVCeleb and LAV-DF, which are both constructed from VoxCeleb2-derived talking-head videos, and the manuscript does not state whether FakeAV test identities or raw clips overlap with LAV-DF training data or vice versa. Additionally, the text describes the FGMDF-matched test list and the 4-second clip-length filter only for the within-database confusion matrix in Fig. 4, not for Table III. If shared identities are present, the reported 83.77/95.44 AUCs could reflect memorization rather than FAU-driven generalization. The authors should report the identity-overlap statistics and specify the exact evaluation lists and clip filters used for every method in Table III.","section":"IV-C / Table III and IV-B"},{"comment":"The text in Section IV-C states that the method outperforms FGMDF by an average binary AUC improvement of 5.74% (90.52% vs. 84.78%), but Table III reports FauForensics's binary average as 89.61 and FGMDF's as 84.78, which is a difference of 4.83 percentage points, and the value 90.52 does not appear in Table III. This unresolved numeric mismatch directly affects the headline 4.83% average improvement claim and must be corrected before the results can be assessed.","section":"IV-C vs. Table III"},{"comment":"The ablation study is reported only for training on LAV-DF and testing on both LAV-DF and FakeAV. The complementary cross-dataset direction, training on FakeAV and testing on LAV-DF, is where Table III shows the largest gain (95.44 AUC), but it is not ablated. Without the complementary direction, it is impossible to attribute the strongest cross-dataset result to the FAU-enhanced feature learning, implicit feature alignment, and temporal attentional pooling modules.","section":"IV-E, Table IV"}],"minor_comments":[{"comment":"The notation in Eq. (1) is difficult to parse because the audio branch applies f_q_at to the audio encoder output while the visual branch applies f_q_vt to the fused FAU-video features; a short table or a more explicit composition would improve readability.","section":"III-A, Eq. (1)"},{"comment":"The claim that the FAU-enhanced visual features are 'dimensionally aligned with the audio latent features Za in R^{T x L}' is not supported because the dimension L is never defined and the relationship between L and the channel dimensions of the audio, video, and FAU encoders is not specified.","section":"III-B"},{"comment":"The caption of Fig. 4 states that all four detectors are evaluated with FGMDF's test list, but the text does not clarify whether the same test list and the same exclusion of shorter videos are used for the numbers in Tables I and II; this should be stated explicitly.","section":"IV-B, Fig. 4"},{"comment":"The phoneme symbol '℧' in Fig. 8 is not standard IPA and should be replaced with the intended phoneme symbol, and the sentence describing the delayed mouth closure should be reworded for clarity.","section":"IV-F, Fig. 8"}],"recommendation":"major_revision","confidential_remarks":"The reviewer's main concern is not the method's plausibility but the lack of protocol transparency in the cross-dataset comparison: the two datasets' shared VoxCeleb2 lineage makes identity leakage a concrete risk, and the current manuscript does not rule it out. The numeric mismatch between the text and Table III should also be fixed before the headline claim is taken at face value. I would encourage the editor to ask for the identity-overlap statistics and exact evaluation protocols rather than rejecting, since the remaining issues appear fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: this is a competent engineering paper with a plausible new combination, but the headline 4.83% cross-dataset gain is not pinned down tightly enough to trust at face value. The architecture is real — FAU features from a frozen ME-GraphAU encoder, frame-wise cross-modal attention, temporal attentional pooling — and the ablations show each component adds something. The within-dataset results are near saturation, so the cross-dataset numbers are the interesting part, and there the paper wobbles.\n\nThe main soft spot is the evaluation protocol. FakeAVCeleb and LAV-DF are both built from VoxCeleb2-derived material, and the paper never reports whether identities or raw clips overlap between the train split of one and the test split of the other. If they do, the model can memorize identities during training and retrieve them at test time, which would inflate the 83.77/95.44 AUCs. The paper also notes FGMDF operates on 4-second clips and excludes shorter videos; a 'fair comparison' using FGMDF's test list appears only in Fig. 4 for the within-database confusion matrix, not in Table III. So the cross-dataset comparison against FGMDF may not be apples-to-apples. There is also a numeric inconsistency: Section IV-C reports 90.52% average binary AUC and a 5.74% gain over FGMDF, while Table III shows 89.61% and a 4.83% gain. One of those is wrong, and it's the number the abstract advertises.\n\nThe method itself also has two under-specified pieces: the fusion operation fφ in Eq. 1 is never defined, and the normalization fnorm in Eq. 4-6 is left as a black box. No code is released. These are not fatal — the ablations still tell a consistent story — but they make the SOTA claim hard to verify.\n\nThe limitations paragraph honestly states that the method assumes audio aligned with the on-screen speaker, which is a real constraint but not a hidden one. The paper knows its scope.\n\nBottom line: worth a serious referee, but the authors should be asked to clarify the cross-dataset protocol, fix the numeric mismatch, and specify or release the missing pieces. If those check out, the 4.83% claim is credible. I'd bring it to a reading group if someone is working on deepfake detection; I'd cite it as related work.","headline":"Solid engineering paper with a plausible FAU-guided audio-visual architecture, but the headline cross-dataset SOTA claim is undercut by an under-specified evaluation protocol and a numeric inconsistency.","tokens_in":17351,"tokens_out":2299,"would_cite":true,"duration_ms":22144,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that facial action units—quantitative descriptors of facial muscle activity—make audio-visual deepfake detection more accurate and more transferable across datasets than prior clip-level methods.","keywords":["deepfake detection","audio-visual forgery","facial action units","cross-dataset generalization","multimodal learning","temporal attention","frame-wise fusion","forensics"],"falsifier":"A decisive experiment is to take the trained model and run the LAV-DF test set with the audio track delayed by about 80–200 milliseconds relative to the video frames. The paper's mechanism predicts a large drop in AUC because frame-wise lip-audio alignment is disrupted; if AUC stays near the reported 99.97, then temporal alignment is not the source of the performance. The same test applied to the cross-dataset setting would show whether the FAU cue alone, without exact alignment, still generalizes.","tokens_in":16367,"feed_emoji":"🎭","tokens_out":9320,"duration_ms":89953,"temperature":0.7,"pith_summary":"This paper tries to establish that a biologically grounded face descriptor—facial action units (FAUs), quantitative readings of facial muscle activity—makes audio-visual deepfake detection more accurate and more transferable across datasets. The authors argue that real videos show stronger temporal correlation in FAU-driven muscle movements than fake videos, so a model that tracks these micro-motion traces should catch manipulations that pixel-level artifacts miss. Their FauForensics pipeline replaces clip-level comparisons with frame-wise audio-visual similarity: a query-shared transformer aligns audio and FAU-enhanced visual features, and a temporal attentional pooler scores intra- and inter-modality consistency. On FakeAVCeleb and LAV-DF they report state-of-the-art within-dataset performance and an average cross-dataset AUC improvement of up to 4.83 points over earlier detectors, plus four-class predictions of which stream was forged.","feed_headline":"Facial muscle cues sharpen deepfake detection across datasets","feed_subtitle":"Fusing facial action units with frame-level lip-audio alignment lifts cross-dataset accuracy by up to 4.83 points.","key_machinery":"The load-bearing object is the facial action unit (FAU), a quantitative descriptor of facial muscle activity such as AU25 for lip parts; the paper treats FAUs as physiologically invariant and therefore harder for generators to fake than raw pixels. The mechanism runs through four stages: FAU-enhanced feature learning fuses features from a frozen FAU encoder with features from a trainable video encoder; implicit feature alignment maps audio and visual latents $\\mathbf{Z}_a, \\mathbf{Z}_v$ into key/value pairs and attends to them with shared learnable queries $\\mathbf{Q} \\in \\mathbb{R}^{T\\times L}$, yielding $\\mathbf{Z}_{aq}$ and $\\mathbf{Z}_{vq}$; a temporal attentional pooler builds dense matrices $\\mathbf{M}_{av}=f_{\\mathrm{norm}}(\\sigma_{av}*f_{\\mathrm{mp}}(\\mathbf{Z}_{aq},\\mathbf{Z}_{vq}))$, $\\mathbf{M}_a$, and $\\mathbf{M}_v$; and separate MLPs turn the flattened matrices into audio, visual, and audio-visual forgery scores. The device converts the physiological premise into an architecture: frame-by-frame lip-audio coordination is scored explicitly, which is what the authors claim transfers across datasets and forgery styles.","core_discovery":"The central claim is that forged audio-visual content breaks the temporal correlation intensity of facial action units: across the more than 20,000 samples examined, real videos exhibit significantly higher FAU consistency than fake ones, making FAUs a forgery-resistant representation tied to facial physiology. The architecture combines a frozen FAU encoder with a trainable video encoder, aligns the two modalities behind shared learnable queries in a transformer, and computes dense frame-by-frame attention matrices for audio-audio, video-video, and audio-video consistency. The authors report state-of-the-art results under both binary and four-class settings on FakeAVCeleb and LAV-DF, with the strongest gains under cross-dataset evaluation. Their ablation attributes the gain to the FAU stream: removing the FAU encoder drops cross-dataset AUC from 83.77 to 78.01 when training on LAV-DF and testing on FakeAVCeleb.","pith_inferences":["The frame-wise attention matrices could be reused as temporal forgery localization heatmaps, a use the paper's design suggests but does not evaluate with localization metrics.","A stress test the authors did not run is dubbed or re-voiced content: when audio is professionally re-synced to a new speaker, the FAU-alignment cue may weaken, so the claimed superiority over other detectors should be re-measured on such data.","Since FAUs are defined by facial anatomy, the same inductive bias could be ported to 3D avatars or synthetic characters whose muscle dynamics are approximated rather than physically generated, extending the detector beyond photorealistic video.","The paper's future direction of phoneme-to-FAU consistency is a natural refinement: aligning frame-level phoneme labels with AU25 motion could localize the exact moment of lip-audio mismatch, which the current dense attention matrix already makes possible."],"forward_implications":["The model outputs separate audio-only, visual-only, and audio-visual scores, so a deployed detector can report which stream was forged rather than a single fake/real label.","Within-database binary performance reaches an average AUC of 99.94 on the two benchmark datasets, matching the strongest prior detector while adding four-class capability.","Cross-dataset binary AUC improves by an average of 4.83 points over the previous state of the art, and the average four-class AUC improvement is larger, indicating the cue transfers rather than memorizing dataset artifacts.","Ablation evidence shows the FAU stream contributes roughly 5.8 points of cross-dataset AUC, so the physiological representation is doing the work, not just the multimodal fusion.","The method keeps high AUC under video/JPEG compression, blur, noise, contrast, and saturation perturbations, supporting its use after realistic post-processing."],"supporting_citations":[{"why":"Provides the LAV-DF benchmark used for within- and cross-database evaluation.","marker":"[53]"},{"why":"Provides the FakeAVCeleb benchmark, including the identity-independent split protocol.","marker":"[54]"},{"why":"The strongest prior multimodal detector it must beat in within- and cross-dataset comparisons.","marker":"[10]"},{"why":"Prior cross-modality regularization method whose evaluation protocol is adopted for FakeAVCeleb.","marker":"[7]"},{"why":"Supplies the pre-trained backbone used as the frozen FAU encoder.","marker":"[50]"},{"why":"Supplies the facial action unit annotations used to pre-train the FAU encoder.","marker":"[52]"},{"why":"Supplies the trainable video encoder backbone.","marker":"[49]"},{"why":"Supplies the audio encoder backbone used to process mel-spectrograms.","marker":"[51]"},{"why":"Supplies the attention mechanism underlying the query-shared multimodal transformer.","marker":"[47]"},{"why":"Supplies the learnable-query design pattern used for implicit cross-modal feature alignment.","marker":"[48]"}],"fun_headline_variants":["FAUs expose deepfake audio-video mismatches","Facial action units boost deepfake detection across datasets","Frame-level lip-audio alignment plus FAUs lift deepfake detection","Deepfake detector leverages facial muscle physiology for cross-dataset gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the audio track is produced by the face visible on screen; when an off-screen speaker, dubbing, or background music provides the audio, the lip-audio correspondence this detector is built to measure is absent, so its inputs stop carrying the signal it relies on.","fun_headline_variants_meta":{"raw":{"variants":["FAUs expose deepfake audio-video mismatches","Facial action units boost deepfake detection across datasets","Frame-level lip-audio alignment plus FAUs lift deepfake detection","Deepfake detector leverages facial muscle physiology for cross-dataset gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000769,"raw_usage":{"total_tokens":3394,"prompt_tokens":920,"completion_tokens":2474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2406}},"tokens_in":536,"tokens_out":2474,"duration_ms":16358,"temperature":1.0,"reasoning_tokens":2406,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:58:21.301378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive experiment is to take the trained model and run the LAV-DF test set with the audio track delayed by about 80–200 milliseconds relative to the video frames. The paper's mechanism predicts a large drop in AUC because frame-wise lip-audio alignment is disrupted; if AUC stays near the reported 99.97, then temporal alignment is not the source of the performance. The same test applied to the cross-dataset setting would show whether the FAU cue alone, without exact alignment, still generalizes.","supporting_citations":[{"cited_title":"Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization,","cited_arxiv_id":null,"evidence_quote":"Provides the LAV-DF benchmark used for within- and cross-database evaluation."},{"cited_title":"Cross- modality and within-modality regularization for audio-visual deepfake detection,","cited_arxiv_id":null,"evidence_quote":"Prior cross-modality regularization method whose evaluation protocol is adopted for FakeAVCeleb."},{"cited_title":"Learning multi- dimensional edge feature-based AU relation graph for facial action unit recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained backbone used as the frozen FAU encoder."},{"cited_title":"Disfa: A spontaneous facial action intensity database,","cited_arxiv_id":null,"evidence_quote":"Supplies the facial action unit annotations used to pre-train the FAU encoder."},{"cited_title":"Video classification with channel-separated convolutional networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the trainable video encoder backbone."}],"review_version":1}