{"id":"f5ff84e8-b3fa-4eae-89ea-3af976429414","arxiv_id":"2412.19104","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Noise-based and masking-based visual pre-training work better together when corruption is inside the encoder, noise is added at lower-layer features, and masked and noised tokens are explicitly separated.","lead":"This paper studies why adding image noise to masked-image-model pre-training has not helped recognition tasks, and identifies three design rules that make it work. The authors build a pre-training method based on those rules and report consistent gains on fine-grained classification, segmentation, and detection benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Disruption-loss and block-2 gains rest on single-run ablations with post hoc selection; reruns with seeds and held-out tasks are needed before the 8.1/8.0 recipe is credible.","rationale":"The reader's weakest_assumption correctly identifies the disruption loss and post hoc block selection as the fragile parts of the argument. I agree that these are the load-bearing elements, but I would sharpen the concern: the entire three-condition recipe is supported only by single-run, bar-chart ablations without variance estimates, and the final configuration is selected from the same benchmarks on which the headline improvements are reported. This makes the 8.1%/8.0% gains vulnerable to selection bias and run-to-run noise, not just to the generalization of the attention-suppression mechanism. I also note that Eq. 9 minimizes row entropy, which is a non-specific sharpening objective; the paper provides no direct evidence that masked tokens are the ones suppressed. These issues are addressable with repeated seeds, held-out design selection, and attention-mass measurements, so they do not warrant rejection, but they do justify the CONDITIONAL verdict. I set verdict_should_be to UNCHANGED because my concern reinforces the reader's conditional assessment rather than moving it to a different verdict.","tokens_in":11997,"tokens_out":4680,"duration_ms":48906,"concrete_test":"Rerun the full pre-training pipeline with 5 independent seeds for: (i) the final method with block 2 + disruption loss, (ii) the same method without Ld, and (iii) the same method with noise injected at blocks 0, 2, 4, and 6. Report mean +/- std fine-tuning accuracy on all benchmarks. Then repeat block selection on a random 50% split of the transfer tasks and evaluate the chosen block only on held-out tasks. If the Ld vs no-Ld gap or the block-2 vs block-4 gap falls below the seed std on most benchmarks, the three-condition recipe is not supported. Additionally, instrument attention rows at the injection block to measure the fraction of attention mass from noisy tokens to masked tokens with and without Ld; if this fraction does not decrease specifically while visible-to-visible attention is preserved, the 'disentanglement' interpretation of Eq. 9 is not validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: three conditions (encoder-style corruption, feature-level noise at a lower block, and disruption loss) together yield the reported gains. The evidence for each condition is bar-chart ablations with no numeric values, no error bars, and no repeated seeds (Figs. 8-12). Critically, the design choices are selected on the same transfer benchmarks used for the final claims: Fig. 9 chooses block 2 by comparing transfer accuracy on those datasets, and Fig. 10 adds the disruption loss after that selection. Under typical ViT-B fine-tuning variance (often 0.3-0.5%), the marginal contributions of feature-level noise and especially the disruption loss may be within run-to-run noise. Furthermore, Eq. 9 minimizes row-wise entropy of the affinity matrix for noisy tokens, but entropy minimization does not by itself guarantee that masked tokens are specifically suppressed; it can sharpen attention onto any token. The paper never reports the actual attention mass assigned to masked tokens, so the 'explicit disentanglement' mechanism is inferred only from aggregate accuracy. If the disruption-loss effect is not significant or acts non-specifically, the third condition collapses and the recipe's advantage is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates why additive-noise-based pre-training has produced only marginal gains on recognition tasks and proposes three design conditions under which noise can help: (1) corruption and restoration should occur inside the encoder, (2) noise should be injected in feature space at a lower encoder block, and (3) masked and noisy tokens should be explicitly disentangled via a disruption loss. Based on these findings, the authors design an encoder-style hybrid masking framework with feature-level noise at block 2 and a disruption loss that minimizes row-wise entropy of the affinity matrix for noisy visible tokens. They report consistent improvements over MIM baselines (MAE, SimMIM) and generative baselines (DiffMAE, MaskDiT) on fine-grained datasets, ImageNet classification, ADE20K segmentation, and COCO detection/segmentation, with claimed gains up to 8.1% over MIM and 8.0% over generative baselines. The evidence is presented almost entirely through bar charts and radar plots in Figures 1-12, with no numerical tables in the main text.","tokens_in":12250,"tokens_out":3447,"duration_ms":33666,"significance":"If the three-condition recipe is validated, the paper would provide a practical and actionable guideline for combining masking and denoising in self-supervised pre-training, addressing a known weakness of MIM on fine-grained, high-frequency tasks. The authors should be credited for running all comparisons under a unified 400-epoch ViT-B setting, re-implementing baselines for fairness, and promising code in the supplementary material. The breadth of downstream tasks is a clear strength. However, the central claim is entirely empirical and currently rests on single-run, post hoc selected ablations without error bars, and the key mechanism (disruption loss) is not directly verified. These are fixable with additional experiments and tables, but until then the quantitative headline claims are not fully supported.","major_comments":[{"comment":"The design choices of noise-injection block (block 2) and the disruption loss are selected on the same transfer benchmarks used for the final claims, and all ablation results are reported as single runs without error bars or multiple seeds. Under typical ViT-B fine-tuning variance of 0.3-0.5%, the marginal gains attributed to feature-level noise and especially to the disruption loss in Figure 10 may lie within run-to-run noise. Please report means and standard deviations over at least three seeds for the ablations and the final comparisons, and ideally hold out a subset of downstream tasks for model selection.","section":"Section 3.4, Figures 9 and 10"},{"comment":"The disruption loss minimizes the row-wise entropy of the affinity matrix rows corresponding to noisy visible tokens. Entropy minimization can sharpen attention onto any token type; it does not by construction guarantee that masked tokens are specifically suppressed, contrary to the text's claim that it 'minimizes the impact of masked tokens xm on noisy visible tokens'. The paper never reports the actual attention mass assigned to masked tokens. Please measure the average attention weight from noisy visible tokens to masked tokens with and without Ld, or modify the loss to explicitly penalize the Avm/Amv entries, so that the disentanglement mechanism is directly verified.","section":"Equation (9), Section 3.4"},{"comment":"All main quantitative results, including the headline 'up to an 8.1% gain over MIM methods and an 8.0% improvement over recent generative baselines', are presented only in figures without exact numerical values, standard deviations, or a comparison table. This makes it impossible for readers to verify individual dataset gains or assess statistical significance. Please include a full numeric table of fine-tuned accuracies (and detection/segmentation metrics) for every method and dataset, with the fine-tuning protocol and number of runs clearly stated.","section":"Section 4.2, Figure 11; Section 1"},{"comment":"The paper concludes that 'the encoder-style approach can indeed outperform the decoder-style in generative pre-training frameworks', but the only direct encoder-style vs decoder-style comparison (Figure 8) shows minimal difference. The later improvements come from adding feature-level noise and the disruption loss to the encoder-style variant only; a decoder-style variant with the same feature-level noise and disruption loss is not tested. Without that controlled comparison, the first condition (corruption and restoration must be applied within the encoder) is not independently supported.","section":"Section 3.3, Figure 8; Section 3.4"}],"minor_comments":[{"comment":"The notation is inconsistent: the input to the MIM framework is called Xmasked = M ⊙ X, but the reconstruction target in Eq. (1) is defined as ¯X = Xvis. Please clarify whether masked positions are zeroed out or removed, and define all symbols consistently.","section":"Equation (1)"},{"comment":"The symbol ˜p is described as 'an element of A' but then indexed as ˜pi,j and required to satisfy P j ˜pi,j = 1, suggesting it is a row-normalized submatrix. Please define whether the loss is applied per head, per layer, and how the row normalization is computed.","section":"Equation (9)"},{"comment":"The 'KL divergence among attention distributions across different heads' is not defined. Please specify the exact formula, the layers and heads included, and the data over which the distributions are averaged.","section":"Figure 2"},{"comment":"The discussion of diffused masking states that 'when the diffusion noise is weak, the pre-training task is trivial', but the noise schedule and timestep sampling distribution used in the comparisons are not given. Please state the schedule and sampling procedure for all variants.","section":"Section 3.2"},{"comment":"The disruption loss is described as 'a variant of masked token optimization proposed in MTO [7]' without specifying the exact modification. Please state how Ld differs from the loss in MTO and why the change is needed for the noisy visible tokens.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The post hoc selection of the noise block on the same benchmarks used for the final claims is the main risk in this manuscript; it is fixable through held-out tasks and multi-seed reporting. The self-citation to MTO [7] is appropriate, but the precise delta from that work should be stated more sharply to avoid novelty concerns. I would encourage requesting the additional controlled experiments before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before you cite it: it claims that noise-based pre-training helps recognition only when (1) corruption and restoration happen inside the encoder, (2) noise is added at the feature level (ideally block 2 of ViT-B), and (3) a disruption loss explicitly suppresses attention between masked and noisy visible tokens. The gains over MAE/SimMIM and DiffMAE/MaskDiT are reported as up to 8.1% and 8.0% across fine-grained and dense tasks.\n\nThe genuinely new part is the combination. Encoder-style corruption plus feature-space noise at a lower block is a natural idea, but I don't know of another paper that puts those together with an explicit task-disentanglement loss. The analysis of why DiffMAE and MaskDiT underperform is also useful: they are decoder-style and inject noise near the decoder, which the authors argue is suboptimal because the encoder is what transfers. That framing is clear and testable.\n\nThe paper is honest in its presentation. The authors re-implement baselines under identical settings, report code in the supplementary material, and explicitly note that a naive encoder-style implementation does not work, which makes their later design choices more credible. The disruption loss is adapted from their own MTO paper, and that lineage is acknowledged.\n\nThat said, the soft spots are real and roughly as the stress-test notes them. The ablations in Figures 8-12 are single-run bar charts with no error bars or seeds, and the best noise block (block 2) is chosen post hoc from the same transfer benchmarks used for the final claims. Under typical ViT-B fine-tuning variance, the marginal contributions of feature-level noise and especially the disruption loss may be within run-to-run noise. The disruption loss minimizes row-wise entropy of the affinity matrix, which sharpens attention but does not by itself prove that masked tokens are specifically suppressed; the paper never reports actual attention mass on masked tokens, so the disentanglement mechanism is inferred only from accuracy. Exact numerical values are also missing from the main text, which makes it hard to judge the size of the effects.\n\nThese are addressable rather than fatal. The recipe is plausible, and the benchmarking across many tasks is a plus. But I would not take the 8.1/8.0 numbers seriously until the authors rerun with multiple seeds, report error bars, and test the disruption-loss mechanism directly (e.g., attention maps with/without the loss, or a held-out set of tasks for model selection).\n\nSend it to peer review. The paper deserves referee time, and the empirical claims are substantial enough that verification is worth the effort. I'd bring it to a reading group to provoke discussion on evaluation practices in SSL.","headline":"A plausible three-part recipe for combining masking and noising in pre-training, with useful analysis but single-run evidence that makes the headline margins unreliable.","tokens_in":12741,"tokens_out":1678,"would_cite":true,"duration_ms":18843,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Noise-based pre-training improves recognition when corruption happens inside the encoder, in feature space, and masked and noisy tokens are explicitly disentangled.","keywords":["masked image modeling","denoising pre-training","generative pre-training","fine-grained recognition","feature-level noise","task disentanglement","disruption loss","self-supervised learning"],"falsifier":"Run the same pre-training with the disruption loss removed but feature-level noise kept at block 2, and check whether fine-grained accuracy on CUB-200-2011 and Stanford Cars drops; if it does not, the disentanglement claim is false. Alternatively, show that adding noise at block 4 or block 6 with the disruption loss matches block 2, which would contradict the claim that lower-layer feature-space injection is critical.","tokens_in":11807,"feed_emoji":"🎯","tokens_out":7689,"duration_ms":60219,"temperature":0.7,"pith_summary":"This paper tries to establish that additive noise, the ingredient behind denoising diffusion models, can improve self-supervised pre-training for recognition, but only when applied in the right place and in the right way. The authors identify three conditions: corruption and restoration must happen inside the encoder that is later transferred, the noise must be added in feature space rather than pixel space, and the network must explicitly stop masked tokens from interfering with noisy visible tokens. Implemented together, these conditions produce a pre-training recipe that outperforms masked image modeling and recent generative baselines on fine-grained classification, ImageNet, semantic segmentation, and detection tasks, with reported gains up to 8.1% over MIM methods and 8.0% over generative baselines. The value of the paper is a concrete architectural guideline for combining masking and denoising in one pre-training framework.","feed_headline":"Noise beats masking for recognition under three conditions","feed_subtitle":"Corruption in the encoder, noise in feature space, and disentangled tokens lift fine-grained and dense task accuracy.","key_machinery":"The carrying mechanism is an encoder-style generative pre-training framework with three coupled components: corruption and restoration inside the encoder, feature-level noise injection, and explicit task disentanglement. The disentanglement is enforced by a disruption loss, a variant of masked token optimization, that acts on the affinity matrix A (the attention-weight map) partitioned into visible-visible, visible-masked, masked-visible, and masked-masked quadrants; it recalibrates the row-wise attention distributions of the noisy visible tokens so that masked tokens have minimal influence on them. The paper also locates the optimal noise injection at encoder block 2 of a ViT-B, arguing that lower layers carry the high-frequency details that fine-grained tasks need.","core_discovery":"The central discovery is that previous attempts to combine denoising with masked image modeling, such as DiffMAE's diffused masking and MaskDiT's hybrid masking, fail to help recognition because they add noise in the wrong place and let the two reconstruction tasks interfere with each other. The paper shows that an encoder-style framework, where both noisy and masked tokens are processed and reconstructed inside the encoder, combined with feature-level noise injection at a lower encoder block and a disruption loss that suppresses attention from masked tokens to noisy visible tokens, captures a broader range of frequency information and transfers better across fine-grained and dense recognition tasks. In the paper's own evaluation, this yields up to an 8.1% gain over MIM methods and an 8.0% improvement over recent generative baselines on CUB-200-2011, NABirds, iNaturalist 2017/2018, Stanford Cars, Aircraft, ImageNet, ADE20K, and COCO.","pith_inferences":["If the frequency-diversity explanation is right, the same three conditions may transfer to other backbone architectures and to video or multi-modal pre-training, though the optimal noise-injection block might shift with the architecture.","The disruption loss can be viewed as a regularizer that prevents the masked-token branch from dominating the shared encoder; a testable variant would replace it with a softer penalty on cross-quadrant attention weights.","The block-2 choice is empirical; one could test whether the optimal block tracks the layer where high-frequency information is most accessible, which would turn the recipe into a predictable rule.","A practical extension is to apply the same conditions to latent diffusion pre-training at larger scale, potentially reducing the data hunger of foundational vision models."],"forward_implications":["Encoder-style corruption and restoration should replace decoder-style placement in generative pre-training aimed at recognition, because the encoder is the part that gets transferred.","Feature-level noise injection is a transferable design choice: adding noise at the input image (pixel space) is measurably worse than adding it at an intermediate encoder block.","When both denoising and de-masking share the encoder, they must be explicitly disentangled; without the disruption loss, the gains largely disappear.","The recipe improves not only fine-grained classification but also ImageNet classification, semantic segmentation on ADE20K, and object and instance segmentation on COCO.","The failure of existing generative pre-training baselines like DiffMAE and MaskDiT on recognition tasks is explained by their violation of these three conditions."],"supporting_citations":[{"why":"MAE is the decoder-style masked autoencoder baseline whose structure the paper contrasts with its encoder-style framework.","marker":"[14]"},{"why":"SimMIM is the encoder-style MIM baseline that the method is compared against and outperformed.","marker":"[40]"},{"why":"DiffMAE is one of the two recent generative pre-training baselines analyzed and shown to give limited recognition gains.","marker":"[39]"},{"why":"MaskDiT is the other recent generative baseline combining masked and noisy tokens; the paper re-evaluates it under identical conditions.","marker":"[42]"},{"why":"MTO supplies the masked token optimization from which the disruption loss is adapted.","marker":"[7]"},{"why":"LDM motivates feature-level (latent-space) noise addition, a pillar of the paper's recipe.","marker":"[31]"},{"why":"This work is cited as evidence for the success of noise applied at the feature level rather than pixel level.","marker":"[6]"}],"fun_headline_variants":["Three conditions make noise boost masked image pre-training","Noise in encoder, feature space, and token separation wins","The right recipe for mixing denoising and masked image modeling","Where to inject noise to make masked image pre-training work"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole recipe rests on the empirical assumption that silencing the attention from masked tokens to noisy visible tokens inside the encoder improves the learned features, an effect the paper supports with ablations that lack error bars and with the noise layer chosen using the same benchmarks on which the final gains are reported.","fun_headline_variants_meta":{"raw":{"variants":["Three conditions make noise boost masked image pre-training","Noise in encoder, feature space, and token separation wins","The right recipe for mixing denoising and masked image modeling","Where to inject noise to make masked image pre-training work"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000829,"raw_usage":{"total_tokens":3586,"prompt_tokens":873,"completion_tokens":2713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2646}},"tokens_in":489,"tokens_out":2713,"duration_ms":162810,"temperature":1.0,"reasoning_tokens":2646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:56:15.284757+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pre-training with the disruption loss removed but feature-level noise kept at block 2, and check whether fine-grained accuracy on CUB-200-2011 and Stanford Cars drops; if it does not, the disentanglement claim is false. Alternatively, show that adding noise at block 4 or block 6 with the disruption loss matches block 2, which would contradict the claim that lower-layer feature-space injection is critical.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"MAE is the decoder-style masked autoencoder baseline whose structure the paper contrasts with its encoder-style framework."},{"cited_title":"Simmim: A simple framework for masked image modeling","cited_arxiv_id":null,"evidence_quote":"SimMIM is the encoder-style MIM baseline that the method is compared against and outperformed."},{"cited_title":"Diffusion models as masked autoencoders","cited_arxiv_id":null,"evidence_quote":"DiffMAE is one of the two recent generative pre-training baselines analyzed and shown to give limited recognition gains."}],"review_version":1}