{"id":"11e7feee-0610-4796-818f-e92b8dc46267","arxiv_id":"2412.07815","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MapDiff, a mask-prior-guided discrete diffusion model for inverse protein folding, achieves state-of-the-art sequence recovery and refoldability on CATH, TS50, and PDB2022 benchmarks.","lead":"A new protein design method, MapDiff, uses a discrete diffusion model with a mask-prior refinement step to generate amino acid sequences from a target 3D backbone. On standard benchmarks it beats previous inverse-folding models, raising median sequence recovery on CATH 4.2 from about 53% to 61%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported CATH/TS50 gains may be inflated by 50-pass Monte-Carlo dropout averaging; baselines are not evaluated at matched inference cost, so the SOTA claim may not reflect the learned denoising architecture.","rationale":"The reader's conditional verdict focuses on posterior-bias accumulation from treating the network's predicted x-hat_0 as the true clean data, which is a legitimate theoretical soft spot but is not the most immediately testable threat to the central empirical claim. In good faith, the more load-bearing concern is whether the reported state-of-the-art results are actually due to the proposed mask-prior-guided denoising architecture or largely to the 50-pass Monte-Carlo dropout ensemble used at inference. The paper itself provides the relevant evidence: Supplementary Fig. 3a shows substantial performance improvement as the number of Monte-Carlo samples grows, with stability only around 20 samples. Since baselines were not evaluated with an equivalent stochastic-averaging procedure, the benchmark comparison does not control for inference budget. Perplexity comparisons are especially susceptible to this confound because averaging many stochastic forward passes reduces the reported perplexity even without improving the model's single-pass predictive distribution. I do not consider this a fatal flaw; the paper clearly discloses its inference settings and releases code, so a matched-compute experiment can settle the issue. For this reason, I keep the reader's CONDITIONAL verdict unchanged rather than escalating to REJECT. I mark agreement as partial because the reader identified a different inference-time vulnerability, while both concerns share the theme that the paper's sampling procedure, rather than the learned mapping alone, carries part of the claimed advantage.","tokens_in":25342,"tokens_out":17552,"duration_ms":184794,"concrete_test":"Using the released MapDiff code, set the Monte-Carlo sample count C=1 and C=2 with all other settings fixed, and recompute median recovery and perplexity on the CATH 4.2 full test set. Compare against GRADE-IF, PiFold, and ProteinMPNN under their default protocols. If MapDiff at C=1 falls below or near GRADE-IF, the SOTA claim depends on ensembling; if it remains above 60% recovery and low perplexity, the concern is resolved. Also report single-pass perplexity without MC averaging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MapDiff's headline results use 50 Monte-Carlo dropout forward passes per DDIM step (Methods, 'Implementation setup'; Algorithm 2), while the four reproduced baselines are evaluated with their default inference protocols. The paper's own sensitivity analysis (Supplementary Fig. 3a) states that recovery 'substantially improves' with the number of Monte-Carlo samples and only stabilizes around 20 samples. Thus Table 1's 61.03% median recovery and perplexity 3.43 characterize a 50-fold stochastic ensemble, not the mask-prior/denoising network by itself. Perplexity is especially confounded: averaging 50 stochastic logit draws can mechanically lower perplexity relative to single-pass baseline evaluations. The central claim 'MapDiff substantially outperforms state-of-the-art' would be weakened if the advantage shrinks or reverses at matched inference budget; the paper provides no such comparison. This is a benchmark-fairness issue distinct from the posterior-bias concern: even if the reverse posterior were exact, the headline comparison as reported does not isolate the method's architectural contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes MapDiff, a discrete denoising diffusion model for inverse protein folding. The method formulates sequence generation as iterative denoising of a categorical amino acid sequence conditioned on a backbone structure, using an EGNN-based base predictor, an entropy-based masking strategy with a mask-ratio adapter, and a pre-trained IPA masked-sequence designer for refinement. Inference combines discrete DDIM step skipping with Monte-Carlo dropout. On CATH 4.2, CATH 4.3, TS50, and PDB2022, the authors report recovery, perplexity, and NSSR improvements over reproduced baselines, plus AlphaFold2 foldability metrics on the CATH 4.2 test set. The paper includes ablations, sensitivity analyses, and qualitative structure comparisons.","tokens_in":25550,"tokens_out":4960,"duration_ms":64044,"significance":"MapDiff is a plausible and well-engineered contribution: the benchmark suite is broad, the main baselines are reproduced, the code and data are publicly archived, and the ablation study decomposes the architecture. The reported AlphaFold2 foldability analysis is an appropriate in-silico check rather than an overclaim. If the headline margins survive matched-inference and uncertainty quantification, the method would be a new state of the art for structure-only inverse folding. However, the central empirical claim is currently supported only by point estimates computed under a substantially larger inference budget than the baselines, so the significance is conditional on the additional comparisons requested below.","major_comments":[{"comment":"The headline comparisons in Table 1 are not made at matched inference cost. MapDiff's reported results use 50 Monte-Carlo dropout samples (C = 50 in Algorithm 2), whereas ProteinMPNN, PiFold, LM-Design, and GRADE-IF are evaluated with their default single-pass settings. The paper's own sensitivity analysis (Supplementary Fig. 3a) shows that recovery 'substantially improves' with the number of Monte-Carlo samples and stabilizes only around 20 samples, so the 61.03% versus 52.63% margin in Table 1 cannot be attributed to the learned denoising architecture alone. Please report MapDiff with a single forward pass, with matched numbers of stochastic samples for stochastic baselines, and at a matched total inference budget such as wall-clock time or FLOPs per protein. Until such a comparison is provided, the 'substantially outperforms' claim is not established.","section":"Implementation setup; Table 1; Supplementary S7/Fig. 3a"},{"comment":"All headline results are single point estimates. None of the recovery, perplexity, NSSR, or foldability comparisons in Tables 1 and 2 include error bars, confidence intervals, or significance tests, although both MapDiff's diffusion sampling and the baselines' decoding are stochastic. Because the central claim is a quantitative margin (e.g., 7.74% recovery on CATH 4.2 and 6.33% NSSR62 on TS50), the authors should report variability across multiple sampling runs or bootstrap resampling over test proteins, and check whether the margins are statistically significant.","section":"Table 1 and Table 2"}],"minor_comments":[{"comment":"The text describes mean-pooling the stochastic logits at each denoising step, but Algorithm 2 as written runs C independent full denoising trajectories and averages only the final-step predictions p^m_0. Please clarify which procedure was actually used, since this affects the interpretation of C and of the sensitivity analysis in Supplementary Fig. 3a.","section":"Algorithm 2 and 'DDIM with Monte-Carlo dropout'"},{"comment":"The checkmark rows in the ablation table are inconsistent with the accompanying text: the text identifies variants 2 and 4 as removing global context and coordinate updating, respectively, but the table's row entries do not clearly show those removals. Please make the component-to-variant mapping explicit so the reader can verify which module deletion produced each result.","section":"Table 3"},{"comment":"The notation [I_k Q_t]^T appears without defining I_k; if this is the identity matrix, the expression should be stated more transparently as x_t Q_t^T, and the typo should be corrected.","section":"Supplementary S3, Eq. (32)"},{"comment":"The reverse posterior substitutes the network prediction x̂_0 for the true clean x_0 in the exact posterior. This is a standard diffusion-model approximation and is acceptable for the empirical claim, but the risk of accumulating bias over the 500-step trajectory should be acknowledged explicitly, since Eq. (8) is presented as an exact derivation.","section":"Methods, Eq. (8) and Supplementary S3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of q-bio.BM and the reproducibility efforts (public code, archived data, reproduced baselines) are genuine strengths. My main concern is the matched-inference issue: the headline advantage may largely reflect the 50-sample Monte-Carlo ensemble rather than the learned denoising network. This is fixable within the manuscript's scope and should not require new biological experiments. I do not see evidence of data leakage or circularity in the benchmark construction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, MapDiff is a genuinely well-built inverse folding paper: discrete diffusion plus entropy masking plus IPA refinement, with code, ablations, and strong numbers on CATH 4.2/4.3, TS50, and PDB2022. Second, the headline 61% vs 52% recovery gap is not apples-to-apples, because MapDiff uses 50 Monte-Carlo dropout forward passes at inference while the reproduced baselines use single-pass evaluation. The paper's own sensitivity analysis (Supp Fig 3a) says recovery improves substantially with MC samples and only stabilizes around 20. So part of the gain is inference budget, not architecture.\n\nWhat's new: the specific combination is new. Discrete diffusion was used in inverse folding by GRADE-IF; entropy-based masking comes from ProRefiner; IPA from AlphaFold/Frame2seq. MapDiff assembles them into a two-step denoiser with a mask-ratio adapter, and the empirical study is careful: standard splits, four datasets, reproduced baselines, ablations showing each component helps, and AlphaFold2 refolding. The transfer gains to TS50 are particularly striking (68% recovery vs 59% for GRADE-IF), though the same inference-budget caveat applies.\n\nThe biggest soft spot is the missing matched-inference-cost comparison. A single-pass MapDiff, or a 50-pass baseline, would tell you how much of the gap is the ensemble rather than the denoising architecture. Second, Table 1 reports single point estimates for the headline metrics; no variance or significance testing across seeds, so we don't know if the 7-8 point recovery gap is stable. Third, the reverse posterior treats the network's predicted x-hat-0 as the true clean data (Eq 8). That's standard for D3PM/DiGress, but it can feed bias back into the 500-step trajectory; Monte-Carlo dropout averages variance but not systematic bias. That concern is real but secondary, since the benchmark result is still informative. The AlphaFold2 foldability analysis is acknowledged in the paper as an in silico proxy rather than experimental confirmation, which is honest.\n\nWho is this for: anyone working on inverse folding or discrete diffusion for protein design. It deserves a serious referee and likely publication after the inference-budget question is addressed. I'd want to see a matched-compute comparison, or at least a single-pass MapDiff number, before taking the SOTA claim at face value.\n\nEngage with it. Send it to review; ask the authors for matched-compute baselines and variance over seeds.","headline":"MapDiff is a real empirical advance in inverse folding, but the headline SOTA gap partly reflects a 50-pass Monte-Carlo inference budget that baselines don't get.","tokens_in":26121,"tokens_out":1907,"would_cite":true,"duration_ms":19328,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A discrete diffusion model with mask-prior refinement sets a new state of the art for inverse protein folding, reaching 61.03% median sequence recovery on CATH 4.2 and the best AlphaFold2 foldability scores without external knowledge.","keywords":["inverse protein folding","discrete denoising diffusion","mask-prior pre-training","protein sequence design","equivariant graph neural network","invariant point attention","sequence recovery","foldability"],"falsifier":"Express a sample of MapDiff-designed sequences — starting with the 1NI8, 2HKY, and 2P0X test cases — and determine their folded structures experimentally; if a large fraction fails to adopt the target folds, the foldability claim is falsified even though the in silico AlphaFold2 metrics look strong.","tokens_in":25162,"feed_emoji":"🧬","tokens_out":9401,"duration_ms":86461,"temperature":0.7,"pith_summary":"The paper claims that inverse protein folding — generating amino acid sequences that fold into a given 3D backbone — is better formulated as a discrete denoising diffusion problem than as a one-shot or autoregressive decoding problem. It introduces MapDiff, a diffusion model that starts from noise over the twenty amino acids and iteratively removes it, conditioned on the backbone structure, using a two-step denoising network. The network first predicts the whole sequence, then masks the low-confidence residues (typically flexible or disordered regions) and refines them with a pre-trained masked sequence designer, so residue interactions carry information to the uncertain positions. Across four benchmarks, MapDiff reports median sequence recovery of 61.03% on CATH 4.2 and 60.86% on CATH 4.3, roughly seven points above previous state of the art, and the best AlphaFold2-refolded similarity (TM-score 88.77%, RMSD 2.57 Å), all without external knowledge such as language models.","feed_headline":"Diffusion model hits 61% recovery on protein design benchmark","feed_subtitle":"Mask-prior denoising also posts the best AlphaFold2 foldability scores on CATH 4.2, with no external knowledge.","key_machinery":"The load-bearing object is the mask-prior-guided denoising network $\\phi_\\theta$, a two-stage denoiser invoked at every reverse-diffusion step. In the first stage, a global-aware equivariant graph neural network (EGNN) maps the noisy sequence and the backbone residue graph to a clean-sequence prediction; in the second, an entropy-based mask with a mask-ratio adapter that scales with the diffusion noise level $\\beta_t$ blanks out the low-confidence residues, and an invariant point attention (IPA) network — pre-trained with BERT-style masked language modelling on the same CATH training data — refines them, the two predictions being blended by entropy-weighted logits. Around this denoiser, the paper builds a discrete denoising diffusion model with transition matrices $Q_t = (1-\\beta_t)I + \\beta_t M$, where $M$ is either the uniform distribution over the 20 amino acids or their marginal distribution in the training data, a cosine noise schedule over 500 steps, and a reverse posterior $q(\\mathbf{x}_{t-1} \\mid \\mathbf{x}_t, \\hat{\\mathbf{x}}_0)$ computed from the network's predicted clean distribution $\\hat{\\mathbf{x}}_0$; discrete DDIM skips steps for speed, and Monte-Carlo dropout averages stochastic forward passes to reduce sampling uncertainty.","core_discovery":"The central discovery is that discrete denoising diffusion with a mask-prior-guided denoising network is a strong generative formulation for inverse protein folding. Concretely, the paper shows that a diffusion process on the 20 amino acid types, conditioned on the backbone residue graph, can be reversed by a network that alternates between a global-equivariant graph predictor (the base sequence predictor) and a pre-trained invariant point attention network that refines exactly the residues the base predictor is least confident about, chosen by an entropy-based mask whose ratio adapts to the noise level of the denoising step. The paper reports that this combination yields median recovery 61.03% and perplexity 3.46 on CATH 4.2 (best prior: 52.63% recovery), similar margins on CATH 4.3, the best zero-shot recovery on TS50 and PDB2022, and the best foldability statistics when the designed sequences are refolded with AlphaFold2, even for sequences with low recovery. The message is that uncertain, low-confidence positions — disordered loops in particular — are better handled by iterative refinement guided by the confidently predicted context than by single-pass or autoregressive prediction, and that this can be achieved without any external knowledge source.","pith_inferences":["The entropy-mask-plus-refinement pattern is portable: if the gains are real, the same base-predictor-then-refine-uncertain-positions design could transfer to antibody loop grafting, small-molecule docking, or RNA design, where confidence also varies sharply across positions.","A testable extension the authors leave implicit is feeding predicted (AlphaFold2-generated) backbones as conditioning input; the mask-ratio adapter might then reveal whether iterative masking also absorbs errors in the structure itself, not only in the sequence.","The contribution of the confidence signal itself, as opposed to the refinement module, could be isolated by comparing entropy-based masking with random masking at equal mask ratios — a control that would make the mechanism fully transparent."],"forward_implications":["Structure-only designers can match or beat models that lean on protein language models and extra training data.","Iterative denoising improves exactly the hard cases: coils, bends, and disordered loops, where the reported margins over prior methods are largest.","Low-confidence residues should be treated as refinement targets; removing the IPA refinement step costs 4.47% recovery in the paper's ablation.","DDIM skipping with Monte-Carlo dropout keeps sequence quality high while generation is accelerated, so speed need not be sacrificed."],"supporting_citations":[{"why":"Supplies the discrete state-space diffusion formalism, including the forward transition matrices and uniform prior that MapDiff builds on.","marker":"[42]"},{"why":"Provides the marginal-transition variant and the discrete graph-generation setting whose posterior sampling MapDiff adapts.","marker":"[34]"},{"why":"The graph-denoising-diffusion baseline for inverse folding that MapDiff extends, and the source of discrete DDIM step-skipping.","marker":"[30]"},{"why":"The main message-passing baseline and the source of the CATH 4.2 dataset split used for training and evaluation.","marker":"[1]"},{"why":"The one-shot non-autoregressive decoding baseline whose global-context and benchmark setup MapDiff draws on.","marker":"[13]"},{"why":"Defines the topology-based CATH 4.3 split and contributes several baseline results the paper compares against.","marker":"[25]"},{"why":"Source of the invariant point attention module and of the AlphaFold2 refolding pipeline used to score foldability.","marker":"[15]"},{"why":"Contributes the entropy-based low-confidence residue selection strategy reused in the mask-prior module.","marker":"[24]"},{"why":"Supplies the implicit denoising scheme that lets MapDiff skip denoising steps during generation.","marker":"[21]"},{"why":"Justifies keeping dropout active at inference to estimate and reduce prediction uncertainty.","marker":"[22]"}],"fun_headline_variants":["Mask-guided diffusion lifts protein recovery to 61 percent","Iterative denoising refines uncertain loops for better protein design","Diffusion beats prior best by 8 points on protein sequence recovery","Uncertainty-aware diffusion tops inverse folding benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reverse step treats the denoising network's predicted clean sequence as if it were the true one when computing the next less-noisy sequence, so a systematic bias in that prediction is fed back and can accumulate across the 500-step trajectory.","fun_headline_variants_meta":{"raw":{"variants":["Mask-guided diffusion lifts protein recovery to 61 percent","Iterative denoising refines uncertain loops for better protein design","Diffusion beats prior best by 8 points on protein sequence recovery","Uncertainty-aware diffusion tops inverse folding benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1402,"prompt_tokens":986,"completion_tokens":416,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":602,"tokens_out":416,"duration_ms":5529,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:55:36.492758+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Express a sample of MapDiff-designed sequences — starting with the 1NI8, 2HKY, and 2P0X test cases — and determine their folded structures experimentally; if a large fraction fails to adopt the target folds, the foldability claim is falsified even though the in silico AlphaFold2 metrics look strong.","supporting_citations":[{"cited_title":"D., Ho, J., Tarlow, D","cited_arxiv_id":null,"evidence_quote":"Supplies the discrete state-space diffusion formalism, including the forward transition matrices and uniform prior that MapDiff builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the marginal-transition variant and the discrete graph-generation setting whose posterior sampling MapDiff adapts."},{"cited_title":"& Wang, Y","cited_arxiv_id":null,"evidence_quote":"The graph-denoising-diffusion baseline for inverse folding that MapDiff extends, and the source of discrete DDIM step-skipping."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The main message-passing baseline and the source of the CATH 4.2 dataset split used for training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The one-shot non-autoregressive decoding baseline whose global-context and benchmark setup MapDiff draws on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the topology-based CATH 4.3 split and contributes several baseline results the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the invariant point attention module and of the AlphaFold2 refolding pipeline used to score foldability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the entropy-based low-confidence residue selection strategy reused in the mask-prior module."},{"cited_title":"& Ermon, S","cited_arxiv_id":null,"evidence_quote":"Supplies the implicit denoising scheme that lets MapDiff skip denoising steps during generation."},{"cited_title":"& Ghahramani, Z","cited_arxiv_id":null,"evidence_quote":"Justifies keeping dropout active at inference to estimate and reduce prediction uncertainty."}],"review_version":1}