{"id":"f535f103-75ce-40e8-9541-53cdee4111d7","arxiv_id":"2505.09484","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MMDA, a CLIP-based multimodal face anti-spoofing framework with differential attention denoising and soft text alignment, reports state-of-the-art cross-domain generalization, but the evaluation is compromised by test-set model selection.","lead":"A face anti-spoofing framework pairs same-domain examples to subtract “noise” from attention and aligns CLIP features to text prompts, reporting near-perfect cross-dataset accuracy on four benchmarks. The results as presented are not trustworthy because the model's exit depth is chosen using test-set labels and no code or error bars are provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set early-exit selection (Sec. 4) directly inflates the reported near-perfect HTER/AUC; a held-out validation selection or fixed-depth protocol is required before the headline generalization claim is credible.","rationale":"The strongest empirical claim is the reported near-perfect cross-dataset performance (avg HTER 4.00%, AUC 98.94% in Table 1). The reader correctly identifies the weakest assumption as honest evaluation. The critical passage is in Section 4 (Implementation Details): 'In the U-DSA module, the depth was set to 7 layers, with the layer yielding the best HTER during testing serving as the early exit point to optimize inference cost.' This sentence states that test-domain HTER is used to pick the early-exit depth. That is test-set model selection. It is not merely a missing error bar; it means the reported numbers are upper envelopes over candidate depths, so they are not unbiased estimates of generalization. This is load-bearing because the entire contribution is an empirical improvement claim, and because the gap to the second-best method is large (12.11% HTER), the selection bias could plausibly explain a substantial portion of it. My concern is therefore a protocol/correctness flaw, not a disagreement with consensus. The paper even highlights the U-DSA depth-dependence in Fig. 4, noting that the best layer varies with the total number of layers, which further emphasizes that early-exit selection is consequential. All secondary issues (missing code, missing lambda value, DADM context) reduce verifiability but are secondary. A concrete re-run with fixed or validation-based early-exit selection, plus repeated seeds, would settle whether the numbers survive. Until then, the headline generalization claim is not supported, and the appropriate disposition is REJECT, matching the reader's verdict exactly.","tokens_in":13487,"tokens_out":2208,"duration_ms":17856,"concrete_test":"Re-run Protocol 1 (CPS→W, CPW→S, CSW→P, PSW→C) under three conditions: (a) fix U-DSA depth at 7 (the maximum, or a depth chosen on the source-domain validation split only), (b) select the early-exit layer using a held-out split of the training domains, and (c) report per-sub-protocol results across at least 5 random seeds with mean and std. If the average HTER under (a) or (b) moves by more than ~3 percentage points, or if the per-seed spread overlaps the second-best method (DADM), the near-perfect headline numbers cannot be attributed to the MMDA components and the central generalization claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that MMDA achieves near-perfect cross-domain generalization. The load-bearing condition is that the reported HTER/AUC are honest estimates of performance on unseen domains. That condition is broken by the implementation detail in Section 4: 'the layer yielding the best HTER during testing serving as the early exit point.' Choosing the U-DSA depth by optimizing HTER on the test domain means the target-domain labels participate in model selection. For a 7-layer module, picking the best of several early-exit points on the test set is a form of test-time model selection; with 12.11% HTER separating MMDA from the previous best, even a modest selection bias across a small set of candidate depths could account for a large fraction of the reported gap. The same concern applies to Table 2 (missing-modality Protocol 2), since the same U-DSA selection mechanism is described globally. If the early-exit depth were instead fixed or chosen on a validation split from the source domains, the reported 4.00% average HTER might degrade substantially; the paper does not report any variance or alternative-protocol numbers. The absence of code and the unreported value of lambda in Eq. (2) compound this, but the test-set selection is the decisive flaw because it directly invalidates the unbiasedness of every headline number. Note this is a correctness/evaluation-flaw critique, not a claim of methodological dishonesty; the issue is that the protocol as written cannot support the empirical conclusion, and the paper's own Section 4 text explicitly flags the selection rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMDA, a CLIP-based framework for multimodal face anti-spoofing that combines a differential-attention denoising module (MD2A), a soft alignment loss to a pre-trained text-defined representation space (RS2), and a U-shaped dual-space adaptation module (U-DSA). The authors report strong cross-dataset generalization results on WMCA, CeFA, PADISI, and SURF under three protocols, with a headline average HTER of 4.00% and AUC of 98.94% under Protocol 1, substantially outperforming prior state of the art. The central claim is that jointly denoising domain and modality noise and softly aligning to a generalized representation space achieves this improvement.","tokens_in":13866,"tokens_out":5981,"duration_ms":59408,"significance":"If the reported results were valid, this would be a meaningful advance for multimodal face anti-spoofing: a 12.11% HTER improvement over the previous best under a standard cross-dataset protocol, plus graceful behavior under missing modalities. The problem of jointly handling domain shift and modality bias is real and the proposed architecture is coherent; the MD2A differential-attention idea and the RS2 soft alignment are reasonable and potentially useful. However, the manuscript provides no code, no error bars, and no value for the key balancing hyperparameter lambda, and — most importantly — the evaluation protocol selects the U-DSA early-exit depth using the test set's HTER. As a result, the reported numbers are not honest estimates of generalization and the central empirical claim is, at present, unsupported.","major_comments":[{"comment":"The U-DSA early-exit depth is selected as 'the layer yielding the best HTER during testing,' which means the test-domain labels participate in model selection among the seven candidate depths. This is a test-set model-selection bias: the reported HTER/AUC values in Tables 1-3 are optimistically biased estimates of generalization, and the headline 4.00% HTER gap over the second-best method could be substantially inflated. The authors must instead fix the depth a priori, select it on a source-domain validation split, or report results for every depth. This issue directly undermines the paper's central claim and must be corrected before the experimental conclusions can be accepted.","section":"Section 4, Implementation Details"},{"comment":"The denoising balance parameter lambda in Eq. (2) is never given a value anywhere in the manuscript, and no sensitivity analysis is reported. Since lambda weights the subtracted noise attention and therefore affects every reported result, omitting it makes the experiments non-reproducible and leaves the robustness of the claimed gains unverified. A value and a sensitivity study (e.g., HTER/AUC vs. lambda over a reasonable range) are required.","section":"Section 3.2, Eq. (2)"},{"comment":"The caption set(s) used for RS2 alignment are not specified in the text. Figure 3 refers to 'ten distinct caption sets,' but the actual captions are never listed. This is not a minor omission: the caption set defines the target representation space into which visual embeddings are aligned, and without it the method cannot be reproduced nor can the influence of this design choice be assessed.","section":"Section 3.3 and Section 4.2"},{"comment":"All results in Tables 1-5 are reported as point estimates without standard deviations, number of runs, or statistical significance tests. In combination with the missing code and the test-set early-exit selection, this makes it impossible to verify whether the observed improvements are reliable. The authors should report mean and standard deviation over multiple runs and make code available.","section":"Section 4"}],"minor_comments":[{"comment":"In the missing-modality discussion, 'the HTER dropping to 93.97%' should be 'the AUC dropping to 93.97%,' and 'at an HTER of 98.30%' should be 'at an AUC of 98.30%.'","section":"Section 4.1"},{"comment":"The full-text title 'Denoising and Alignment: Rethinking Domain Generalization for Multimodal Face Anti-Spoofing' differs from the arXiv title 'Purify then Guide: Rethinking Domain Generalization for Multimodal Face Anti-Spoofing.' Please align them.","section":"Title"},{"comment":"The notation in Eq. (7) for U-DSA is confusing: the definitions of v_i, v'_i, Adapt, and Remap, and the roles of indices i and d, should be spelled out more carefully to make the residual remapping unambiguous.","section":"Section 3.2, Eq. (7)"},{"comment":"The text says the batch reorganization 'randomly selects' same-domain samples, but the algorithm takes the first same-domain sample in the batch. Please clarify whether the selection is random or deterministic.","section":"Algorithm 1"},{"comment":"Figure 3 shows layer depths up to 11, while Section 4 states the U-DSA depth is 7; Figure 4 shows total layer numbers from 1 to 7. These numbers should be reconciled or explicitly explained.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The test-set early-exit selection in Section 4 is a serious protocol flaw that invalidates the reported numbers as estimates of generalization. I recommend major revision rather than immediate rejection only because the flaw is in principle fixable by re-running the experiments with a proper validation-based depth selection. If the corrected results do not preserve the reported large gains, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a case where a clean idea meets a broken ruler. The proposed MMDA framework is a reasonable assembly of known pieces—differential attention, CLIP text alignment, adapters—with one genuinely interesting twist: pairing samples within the same domain to define a 'noise' attention term. If the reported numbers were trustworthy, the 12-point HTER drop would be a big deal in face anti-spoofing. They're not, because the paper selects the U-DSA early-exit layer based on test-set HTER (Sec 4 Implementation Details). That makes the test labels part of model selection, so the headline 4.00% HTER is not an unbiased generalization estimate. The same issue runs through all three protocols and the ablations.\n\nWhat's genuinely new: the MD2A denoising formulation, using same-domain pairs to extract a noise attention and subtracting it (Eq. 2), is a neat extension of differential attention to domain/modality noise. The RS2 soft alignment and the U-shaped remapping are plausible and reasonably motivated. The ablations in Tables 4/5 are consistent with the components helping, but they suffer from the same early-exit selection, so they should be taken as suggestive only. The paper is clearly written and the related work is honestly positioned.\n\nWhere it falls apart: the early-exit selection is not a minor implementation detail. Picking the best of several depths on the test domain can easily eat a large fraction of a 12-point gap, especially when the other methods were evaluated with fixed architectures. There is also no reported value for lambda in Eq. (2), no code, and no multiple-seed variance. The comparison to DADM, a closely related method with overlapping authors, is fine to include but should say how the setups differ. These are fixable problems. The right experiment is to fix the early-exit depth (or choose it on a source-domain validation split), report lambda and seeds, and release code.\n\nBottom line: the framework is worth taking seriously, but the empirical claim as stated is not supported. I'd send this to review—not desk reject—because the flaw is identifiable and correctable, and the method itself deserves scrutiny. The authors should be required to redo the evaluation without test-set model selection.","headline":"Interesting framework with a clever denoising idea, but test-set early-exit selection invalidates the headline numbers as they stand.","tokens_in":14366,"tokens_out":2618,"would_cite":false,"duration_ms":25256,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MMDA reports near-perfect cross-domain generalization for multimodal face anti-spoofing by denoising and aligning into a frozen CLIP space.","keywords":["multimodal face anti-spoofing","domain generalization","presentation attack detection","CLIP","differential attention","representation-space alignment","modality bias","parameter-efficient transfer learning"],"falsifier":"Re-run the Protocol 1 evaluation with the U-DSA early-exit depth fixed before any test labels are seen, for example always using layer 7 or selecting the depth on a held-out validation domain. If the average HTER rises materially above 4.00%, the near-perfect result is inflated by test-set model selection; if it stays near 4.00%, the central claim survives.","tokens_in":13209,"feed_emoji":"🛡️","tokens_out":8752,"duration_ms":81720,"temperature":0.7,"pith_summary":"The paper tries to establish that multimodal face anti-spoofing can generalize to unseen domains when the model suppresses two kinds of noise at once—the bias added by each sensor modality and the shift added by each capture domain—and keeps its features inside the generalized representation space of a frozen pre-trained CLIP model. It argues that prior methods collapse these two problems into one, so their decision boundaries overfit. The proposed MMDA framework performs denoising through a differential-attention module, soft alignment of visual embeddings to CLIP text embeddings, and U-shaped remapping that preserves the pre-trained space. On four datasets under the MMDG protocol the paper reports an average HTER of 4.00% and AUC of 98.94%, beating the previous best by 12.11% HTER. If those numbers survive a fair evaluation, the practical consequence is that cross-dataset spoofing detection can reach near-perfect accuracy with only lightweight adapters on a frozen CLIP backbone.","feed_headline":"Spoof detector hits 4% error on unseen face datasets","feed_subtitle":"MMDA removes domain and modality noise, preserves a pre-trained representation space, and nearly perfects cross-dataset anti-spoofing.","key_machinery":"The load-bearing object is the Modality-Domain Joint Differential Attention (MD2A) module, which replaces a single attention map with the difference of two softmax attention maps: $\\mathrm{softmax}(QK^\\top/\\sqrt{n_d}) - \\lambda \\mathrm{softmax}(Q'K'^\\top/\\sqrt{n_d})$, where the second map is computed from a paired sample drawn from the same domain and is intended to capture common domain and modality noise. Subtracting it suppresses noise before the values are aggregated. Two supporting mechanisms carry the alignment part: the Representation Space Soft (RS2) alignment, which minimizes the cosine distance from each visual embedding to its nearest CLIP text embedding while also applying a classification loss, and the U-shaped Dual Space Adaptation (U-DSA) module, which uses residual MLPs to remap deeper adapted features back to shallower pre-trained space so that fine-tuning depth does not destroy the frozen space's generalization.","core_discovery":"On the paper's own terms, the central discovery is that domain generalization for multimodal face anti-spoofing reduces to two solvable subproblems: removing the combined domain and modality noise before fusion, and aligning multimodal data into a pre-trained, text-anchored representation space instead of learning a brand-new decision boundary. The MMDA pipeline implements this with a Modality-Domain Joint Differential Attention module that subtracts a same-domain noise attention map from the ordinary attention map, a Representation Space Soft alignment loss that pulls each visual embedding toward its nearest CLIP text embedding without forcing hard visual-text correspondence, and a U-shaped Dual Space Adaptation module that remaps deep adapted features back to the shallow pre-trained space. The paper reports that this reaches an average HTER of 4.00% and AUC of 98.94% under Protocol 1 cross-dataset testing, and 10.14% average HTER when modalities are missing at test time. The authors take these results as confirmation that denoising and representation-space alignment address the fundamental difficulty of multimodal domain generalization, rather than just squeezing more accuracy out of a known benchmark.","pith_inferences":["We infer that the reported Protocol 1 numbers are likely optimistic: Section 4 says the early-exit layer is chosen as the one yielding the best HTER during testing, which means test labels participate in model selection; a fair evaluation would fix the depth before seeing the test set.","The near-perfect scores suggest these four benchmarks may be close to saturation for CLIP-based face anti-spoofing, so distinguishing future methods will require harder protocols with more domain diversity or unseen attack types.","The same recipe—same-domain noise subtraction plus soft alignment into a frozen text-anchored space—could be transferred to other multimodal domain-generalization tasks such as RGB-depth scene recognition, by keeping the paired-same-domain differential attention and the nearest-text-embedding soft loss."],"forward_implications":["Under Protocol 1, MMDA is reported to reach 4.00% average HTER and 98.94% AUC across the four unseen target datasets, cutting the previous best HTER by 12.11 percentage points.","Under Protocol 2, with one or two modalities missing at test time, MMDA still reports 10.14% average HTER without any dropout-specific training, with RGB and IR largely compensating for each other.","Under Protocol 3, training on only two source domains yields 7.52% and 6.30% HTER on the two transfer directions, suggesting the method does not need many domains to generalize.","Ablations attribute the gains to the design: MD2A lowers HTER from 22.92% to 9.70% on the MoE adapter setting, and adding RS2 lowers it further to 8.88%."],"supporting_citations":[{"why":"Supplies the MMDG evaluation protocol, including Protocols 1-3 and the four benchmark datasets, which the central empirical claims are measured against.","marker":"[21]"},{"why":"Provides the frozen CLIP image and text encoders whose zero-shot generalized representation space the alignment strategies build on.","marker":"[29]"},{"why":"Introduces differential attention, the mechanism MD2A extends to subtract domain and modality noise.","marker":"[37]"},{"why":"Frames multimodal domain generalization as a noise and alignment problem and motivates why unimodal DG methods fail when transferred to multimodal FAS.","marker":"[5]"},{"why":"Is the strongest previous baseline in Protocol 1; the 12.11% HTER gap is measured against it.","marker":"[36]"},{"why":"Provides a CLIP-based, language-guided FAS baseline that shows the starting point for fine-tuning a frozen CLIP backbone.","marker":"[31]"},{"why":"Supplies the S-Adapter PETL baseline used in comparisons, representing parameter-efficient adapters for FAS.","marker":"[4]"}],"fun_headline_variants":["MMDA framework: 4% HTER on unseen face anti-spoofing","Denoise and align to CLIP: cross-domain anti-spoofing wins","Joint noise removal and CLIP alignment beat cross-domain spoofing","Purify then guide: rethinking multimodal face anti-spoofing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported cross-domain error rates are honest, unbiased estimates of performance on unseen domains; the paper's own implementation detail—choosing the U-DSA early-exit layer by the best HTER during testing—makes that premise doubtful.","fun_headline_variants_meta":{"raw":{"variants":["MMDA framework: 4% HTER on unseen face anti-spoofing","Denoise and align to CLIP: cross-domain anti-spoofing wins","Joint noise removal and CLIP alignment beat cross-domain spoofing","Purify then guide: rethinking multimodal face anti-spoofing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000427,"raw_usage":{"total_tokens":2265,"prompt_tokens":1107,"completion_tokens":1158,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":1078}},"tokens_in":723,"tokens_out":1158,"duration_ms":11241,"temperature":1.0,"reasoning_tokens":1078,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:29:27.279062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Protocol 1 evaluation with the U-DSA early-exit depth fixed before any test labels are seen, for example always using layer 7 or selecting the depth on a held-out validation domain. If the average HTER rises materially above 4.00%, the near-perfect result is inflated by test-set model selection; if it stays near 4.00%, the central claim survives.","supporting_citations":[{"cited_title":"Suppress and rebalance: Towards generalized multi-modal face anti-spoofing","cited_arxiv_id":null,"evidence_quote":"Supplies the MMDG evaluation protocol, including Protocols 1-3 and the four benchmark datasets, which the central empirical claims are measured against."},{"cited_title":"Simmmdg: A simple and effective framework for multi-modal domain generalization","cited_arxiv_id":null,"evidence_quote":"Frames multimodal domain generalization as a noise and alignment problem and motivates why unimodal DG methods fail when transferred to multimodal FAS."},{"cited_title":"DADM: Dual Alignment of Domain and Modality for Face Anti-spoofing","cited_arxiv_id":"2503.00429","evidence_quote":"Is the strongest previous baseline in Protocol 1; the 12.11% HTER gap is measured against it."},{"cited_title":"Flip: Cross-domain face anti- spoofing with language guidance","cited_arxiv_id":null,"evidence_quote":"Provides a CLIP-based, language-guided FAS baseline that shows the starting point for fine-tuning a frozen CLIP backbone."},{"cited_title":"S-adapter: Generalizing vision transformer for face anti-spoofing with statistical tokens","cited_arxiv_id":null,"evidence_quote":"Supplies the S-Adapter PETL baseline used in comparisons, representing parameter-efficient adapters for FAS."}],"review_version":1}