{"id":"845718e4-0738-43a9-aa06-e28e18b93483","arxiv_id":"2501.08137","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A detector that scores per-frame audio-visual timing mismatches, trained with pseudo-fakes edited locally in time, beats prior audio-visual deepfake detectors on DFDC and FakeAVCeleb.","lead":"This paper detects audio-visual deepfakes by measuring how well audio and video match at every moment in time, and by training on videos with artificially introduced local timing glitches. It reports top accuracy on two public deepfake benchmarks, suggesting that timing mismatches are a useful clue for catching fakes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA numbers are selected on the same test sets used for comparison; the claimed superiority is not independently established without a validation split.","rationale":"The reader's verdict is CONDITIONAL, and my concern reinforces that conditionality rather than changing it. My chosen concern differs from the reader's weakest assumption: the reader focused on whether pseudo-fake temporal edits are a faithful proxy for real deepfake artifacts, while I focus on the more immediate protocol issue that the final configuration appears to have been selected using the same test sets later used for the SOTA comparison. This is load-bearing because the central claim is empirical superiority; if a proper validation split were used and the chosen configuration changed, the reported AUC margins could shrink or disappear. The concern is concrete and checkable: Section 3.2.1 and Section 3.2.3 explicitly evaluate augmentation variants and temporal sizes on the in-dataset and cross-dataset test sets, and Section 3.3 then adopts the 'optimal setup' from those comparisons. No validation set is mentioned, and the checkpoint selection criterion is lowest training loss, which does not provide an unbiased model-selection signal. I am not claiming the method is invalid or that the authors acted improperly; the issue is that the reported numbers cannot, as presented, support the claim of surpassing prior work. The reader's proxy concern is also legitimate, but it is harder to settle definitively without a new dataset or artifact-level analysis; the validation-split test is a direct, feasible check that would settle the protocol concern. Therefore the verdict should remain CONDITIONAL, with the condition being a proper validation-based selection and uncertainty quantification of the reported AUC differences.","tokens_in":6288,"tokens_out":5019,"duration_ms":54551,"concrete_test":"Hold out a validation split from the DFDC training set (e.g., 10% of the 15,300 videos, stratified by real/fake). Re-run the Table 1 and Table 2 ablations using only validation AUC to select the augmentation type and T', then evaluate that single chosen configuration once on the DFDC test set and once on FakeAVCeleb. If the selected configuration differs, or if the test AUC is not above FGI's 97.7%/84.5% by more than the bootstrap or DeLong confidence interval, the paper's SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The SOTA comparison in Section 3.3 uses the 'optimal setup (T'=7 with attention and clip replacement)', but that setup was chosen from the same test sets on which the comparison is reported. Section 3.2.1 selects clip-replacement augmentation by comparing Table 1 rows on the DFDC test set and the FakeAVCeleb cross-dataset test set; Section 3.2.3 selects T'=7 by comparing Table 2 rows on those same sets. No held-out validation split is described; the implementation details only mention selecting the checkpoint by lowest training loss. Because every candidate configuration was scored on the test data before the final numbers were reported, the 98.0% and 87.0% AUC values are selected-test estimates rather than independent evaluations. The claimed margin over FGI (97.7% in-dataset, 84.5% cross-dataset) therefore does not follow from these tables. This is a protocol flaw independent of whether the pseudo-fake temporal edits faithfully model real deepfake artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an audio-visual deepfake detector that combines a temporal distance map between audio and visual features, an attention mechanism over time steps, and pseudo-fake data augmentation based on local temporal manipulations (clip replacement, repetition, flipping, and translation). The method is trained on a DFDC subset and evaluated in-dataset on DFDC and cross-dataset on FakeAVCeleb, reporting video-level AUC of 98.0% and 87.0%, respectively, with the configuration T'=7 plus attention and clip-replacement augmentation.","tokens_in":6491,"tokens_out":1782,"duration_ms":18970,"significance":"If the reported results are valid, the paper makes a useful contribution by shifting fine-grained inconsistency modeling from the spatial to the temporal domain and by exploring several local temporal augmentation strategies. The architecture is simple, the augmentation idea is clearly described, and the cross-dataset evaluation direction is appropriate. However, the central SOTA claim is currently undermined by a test-set selection protocol: the augmentation type and T' value are chosen from the same test sets on which the final comparison is reported. The cross-dataset evidence is also based on only 140 videos without uncertainty quantification. These issues are fixable, but they must be addressed before the claimed superiority over prior methods can be accepted.","major_comments":[{"comment":"The configuration used for the SOTA comparison, namely clip-replacement augmentation and T'=7, is selected by comparing rows of Table 1 and Table 2 on the DFDC test set and the FakeAVCeleb cross-dataset test set. No held-out validation split is described; the implementation details mention only checkpoint selection by lowest training loss. Consequently, the reported AUC values of 98.0% and 87.0% in Tables 3 and 4 are selected-test estimates rather than independent evaluations, and the claimed margins over FGI (97.7% and 84.5%) do not follow. Please introduce a proper validation split for model selection, or use nested cross-validation, and report the performance of the selected configuration on a truly held-out test set.","section":"Section 3.2.1 and Section 3.2.3, Tables 1 and 2"},{"comment":"The cross-dataset evaluation uses only 70 real and 70 fake videos, and no error bars, confidence intervals, or significance tests are reported. A 2.5 AUC point difference over FGI (87.0% vs 84.5%) on such a small set may be within sampling variability. Please report per-video bootstrap confidence intervals or results over multiple training seeds, and state whether the difference is statistically significant.","section":"Section 3.1, FakeAVCeleb cross-dataset evaluation"},{"comment":"The pseudo-fake examples are generated by local temporal edits (replacing, repeating, flipping, translating) and the classifier is trained only on these synthetic inconsistencies. The paper does not provide evidence that these edits are a faithful proxy for the temporal artifacts produced by real deepfake pipelines. Please analyze where the model attends in real fakes, or evaluate on real forged videos with known temporal manipulation, to support the claim that the method detects actual temporal inconsistencies rather than synthetic edit patterns.","section":"Section 2.1.2 and Section 3.2.1"}],"minor_comments":[{"comment":"The dataset name is inconsistently typeset as 'FakeA VCeleb' in the abstract and 'FakeAVCeleb' elsewhere; please unify.","section":"Abstract and full text"},{"comment":"The indexing in the repeating formula C = A_{i+⌊a/p⌋*p} is unclear; it should be made explicit that each element in the chunk is drawn from the position i + p*⌊a/p⌋, or be rewritten with clearer notation.","section":"Section 2.1.2, Eq. (3)"},{"comment":"There is a typo 'we uses f = 2' in the flipping example; also, the caption lists six subfigures (a)-(f) but the text describes five manipulation types, which may confuse readers.","section":"Fig. 2 caption and Section 2.1.2"},{"comment":"No code or trained models are mentioned as being released; providing code would strengthen reproducibility of the augmentation and attention implementation.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper shares authors and ideas with prior work by the same group (reference [9]), and the current manuscript builds directly on it; the editor may wish to verify that the novelty over [9] is sufficiently clear to reviewers and readers. The main technical concern, test-set selection, is a protocol issue that is fixable with additional experiments, so I do not recommend rejection, but the revised version must either provide a proper validation protocol or temper the SOTA claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nYou asked about the Astrid et al. audio-visual deepfake detection paper. Bottom line: there is a real nugget here — a temporal distance map that compares audio and visual features at each time step, plus an attention mechanism over time, and a set of simple local temporal pseudo-fake augmentations (repeat, flip, translate). As far as I can tell, the temporal fine-grained treatment is new, and the augmentations are a nice extension of their own spatial fine-grained predecessor [9]. The cross-dataset result on FakeAVCeleb is the strongest selling point: 87% AUC vs 84.5% for FGI is a meaningful gap, assuming it holds up.\n\nAnd there's the rub. The paper selects its best configuration (T'=7, attention, clip replacement) using test-set AUC from both the in-dataset and cross-dataset tables. No held-out validation split is described. That means the reported 98.0% and 87.0% are the best of several configurations looked at on the same test sets, not independent evaluations. The in-dataset gap over FGI is only 0.3 points (98.0 vs 97.7), which is almost certainly within noise — no error bars or significance tests anywhere. The cross-dataset gap is larger (2.5 points) but sits on only 140 videos, with no variance estimate. The stress-test note is correct: the claimed superiority is not independently established from these tables.\n\nThe pseudo-fake proxy assumption is also worth flagging. The paper trains on locally edited sequences (repeating, flipping, translating chunks) as if they represent real deepfake temporal artifacts. The replacement augmentation is plausible; the others are more speculative. The cross-dataset improvement suggests some generalization, but it's not a direct validation on real forgeries.\n\nThat said, the method is clearly described, the ablations are sensible, and the idea is honest. The protocol flaw is fixable: use a validation split for hyperparameter selection, train several seeds, report mean/variance. The paper would be more credible after that. As is, it's a solid workshop/conf submission with a promising direction, not a proven SOTA claim.\n\nI'd send it to review — a serious referee will catch the selection issue and the small test set, and the authors can address it. Worth a reading group discussion on test-set selection in deepfake detection, maybe.\n\nBest.","headline":"Promising temporal fine-grained AV deepfake detector, but the reported SOTA margins are selected on the test set and the cross-dataset claim rests on 140 videos.","tokens_in":6991,"tokens_out":3373,"would_cite":false,"duration_ms":31885,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that audio-visual deepfakes can be caught by measuring fine-grained temporal mismatches between the audio and visual streams at every time step, rather than relying on a single global consistency score.","keywords":["audio-visual deepfake detection","temporal inconsistency","fine-grained classification","pseudo-fake augmentation","attention mechanism","DFDC","FakeAVCeleb","cross-dataset generalization"],"falsifier":"Train the reported model on the DFDC subset and apply it to a set of real deepfakes whose only manipulation is a full audio-track replacement or a constant audio delay; if the AUC falls to the level of the no-augmentation baseline, the local temporal-edit proxy is the reason the cross-dataset result does not transfer.","tokens_in":6101,"feed_emoji":"🎭","tokens_out":6360,"duration_ms":60630,"temperature":0.7,"pith_summary":"This paper claims that audio-visual deepfakes leave small timing errors between the sound and image tracks, and that a detector can catch them by comparing the two streams at every moment instead of using one global score. To do this, the authors introduce a temporal distance map, meaning the per-time-step L2 distance between visual and audio features, plus an attention mechanism that down-weights irrelevant moments such as background sound. They also train on pseudo-fakes built by locally reordering, repeating, flipping, or translating short chunks of a clip, so the model learns fine-grained temporal inconsistencies. With the best configuration, the reported AUC is 98.0% on the DFDC test set and 87.0% when transferred to FakeAVCeleb, both above the compared state-of-the-art methods.","feed_headline":"Per-frame audio-video mismatch flags deepfakes","feed_subtitle":"Temporal distance maps plus local fake edits reach 98% AUC in-dataset and 87% cross-dataset.","key_machinery":"The central mechanism is the temporal distance map $m_t = \\lVert f^v_t - f^a_t \\rVert$, computed from a shallow 3D convolutional visual encoder and a 1D convolutional audio encoder aligned to the same $T'$ time steps. A cross-attention map weighs each time step by the correlation between projected audio and visual features, and the classifier consumes the attended map $\\hat{m} = m \\odot a$. Training also relies on pseudo-fakes generated by local temporal manipulations over a random contiguous sub-sequence: replacement with another clip, frame repetition, flipping, and left/right translation, so the model sees subtle temporal inconsistencies instead of only global audio-visual mismatches.","core_discovery":"The paper's central claim is that temporal fine-grained inconsistencies are a usable forensic signal for audio-visual deepfakes. The proposed representation is a distance vector $m$ of length $T'$, where each entry $m_t$ is the L2 distance between the visual feature and the audio feature at time step $t$, multiplied element-wise by a softmax attention map before classification. The authors report that this local distance map outperforms both a single global distance ($T'=1$) and a longer map ($T'=15$), and that their pseudo-fake augmentation, especially replacing a local sub-sequence with a chunk from another clip, gives the largest cross-dataset gain.","pith_inferences":["The authors do not test partial or localized forgeries; a natural extension is to use a threshold on the attended distance map to identify which temporal segments in a video are fake.","Because temporal local distance outperforms the spatial local distance of the prior method they extend, combining both temporal and spatial fine-grained maps could be more robust than either alone.","The synthetic edits cover only simple timing distortions; testing the same training recipe on fakes with continuous audio delay or pitch shifts would reveal whether the proxy captures a broader class of real artifacts."],"forward_implications":["A detector can score audio-visual alignment at every time step and flag fakes without identity-specific training or spatial artifact cues.","Using $T'=7$ time steps outperforms both a single global distance ($T'=1$) and a longer map ($T'=15$), so the useful signal is local but not overly fine.","Attention over the temporal distance map is what lifts cross-dataset AUC on FakeAVCeleb from 72.14% to 80.34%.","Replacing a short sub-sequence with a chunk from a different clip is the most effective pseudo-fake operation, suggesting that content substitution is the artifact class that generalizes best.","With the chosen configuration, the reported AUCs of 98.0% (DFDC) and 87.0% (FakeAVCeleb) exceed all compared state-of-the-art methods in the paper."],"supporting_citations":[{"why":"The prior fine-grained audio-visual detector this work extends; contributes the replacement-based pseudo-fake idea and the attention design adapted here to time.","marker":"[9]"},{"why":"Supplies the dissonance-based detection baseline and the data preprocessing protocol used for DFDC.","marker":"[3]"},{"why":"Provides the DFDC dataset used for training and in-dataset evaluation.","marker":"[15]"},{"why":"Provides the FakeAVCeleb dataset used for cross-dataset evaluation.","marker":"[16]"},{"why":"Defines the cross-dataset protocol of 70 real and 70 fake FakeAVCeleb videos.","marker":"[18]"},{"why":"A state-of-the-art audio-visual baseline with 96.7% in-dataset AUC that the method claims to surpass.","marker":"[6]"},{"why":"A baseline whose cross-dataset AUC of 82.8% is the strongest previous transfer result the method claims to beat.","marker":"[10]"}],"fun_headline_variants":["Local time gaps reveal deepfake audio-video","Sync slips in audio-video spot deepfakes","Micro sync errors unmask deepfakes","Audio-video temporal glitches flag fakes","Precise timing mismatches catch deepfakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that pseudo-fakes created by locally editing a real clip, namely repeating, flipping, translating, or swapping a few frames, faithfully mimic the timing errors found in real deepfakes, so a detector trained only on those edits will transfer to real forgeries.","fun_headline_variants_meta":{"raw":{"variants":["Local time gaps reveal deepfake audio-video","Sync slips in audio-video spot deepfakes","Micro sync errors unmask deepfakes","Audio-video temporal glitches flag fakes","Precise timing mismatches catch deepfakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000395,"raw_usage":{"total_tokens":1985,"prompt_tokens":769,"completion_tokens":1216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":385,"completion_tokens_details":{"reasoning_tokens":1144}},"tokens_in":385,"tokens_out":1216,"duration_ms":10739,"temperature":1.0,"reasoning_tokens":1144,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:28:48.444767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the reported model on the DFDC subset and apply it to a set of real deepfakes whose only manipulation is a full audio-track replacement or a constant audio delay; if the AUC falls to the level of the no-augmentation baseline, the local temporal-edit proxy is the reason the cross-dataset result does not transfer.","supporting_citations":[{"cited_title":"Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’,","cited_arxiv_id":null,"evidence_quote":"The prior fine-grained audio-visual detector this work extends; contributes the replacement-based pseudo-fake idea and the attention design adapted here to time."},{"cited_title":"This method, depicted in Fig","cited_arxiv_id":null,"evidence_quote":"Supplies the dissonance-based detection baseline and the data preprocessing protocol used for DFDC."},{"cited_title":"Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake detection,","cited_arxiv_id":null,"evidence_quote":"Provides the DFDC dataset used for training and in-dataset evaluation."},{"cited_title":"Multi- attentional deepfake detection,","cited_arxiv_id":null,"evidence_quote":"Provides the FakeAVCeleb dataset used for cross-dataset evaluation."},{"cited_title":"Avoid-df: Audio-visual joint learning for detecting deepfake,","cited_arxiv_id":null,"evidence_quote":"Defines the cross-dataset protocol of 70 real and 70 fake FakeAVCeleb videos."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A state-of-the-art audio-visual baseline with 96.7% in-dataset AUC that the method claims to surpass."},{"cited_title":"Deepfake video targeting zelen- sky’s wife linked to russian disinformation campaign, cnn analysis shows,","cited_arxiv_id":null,"evidence_quote":"A baseline whose cross-dataset AUC of 82.8% is the strongest previous transfer result the method claims to beat."}],"review_version":1}