{"id":"7f154a45-7cd4-483d-b7ce-3579204ac4a4","arxiv_id":"2506.13485","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RefineNovo, a non-autoregressive peptide sequencing model with CTC-based curriculum masking and iterative refinement, reports state-of-the-art amino acid precision and peptide recall on the 9-species benchmarks.","lead":"This paper adds a curriculum learning step to non-autoregressive Transformer models for de novo peptide sequencing, masking parts of the peptide target so the model learns easy examples before hard ones. The method reports better sequencing accuracy on nine-species benchmarks and claims more stable training than the prior NAT model PrimeNovo.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RefineNovo's iterative refinement is trained on oracle CTC paths but applied to the model's own argmax path; no experiment tests this transfer, and the reported gain (0.006 AA precision) is too small to trust without error bars.","rationale":"The reader's weakest assumption is exactly the point I find load-bearing: Sections 3.3 and 3.4 train on oracle paths and infer on argmax paths. I agree with that identification. I selected it over the missing ContraNovo baseline, missing error bars, and the ambiguously defined 90% failure-reduction experiment because even if those reporting gaps were fixed, the paper's central methodological claim—that learned curriculum embeddings can be reused for iterative self-refinement—would still rest on an untested distribution transfer. The paper does provide some internal evidence: Table 5 shows monotonic improvement with iterations, Table 4 shows the full model beating the baseline, and Appendix E adds NovoBench results. Those are genuine but limited: all are single-run and the refinement gain is 0.006–0.008, an order of magnitude smaller than the species-to-species variance in Tables 1 and 2. A conditional accept with a required oracle-vs-self-conditioned comparison is the right verdict; if that comparison fails, the refinement module should be removed from the SOTA claim and the remaining curriculum contribution would need its own statistical support.","tokens_in":21562,"tokens_out":7540,"duration_ms":77385,"concrete_test":"Run the released RefineNovo inference on 9-species-V1 and V2 under three settings with decoding otherwise fixed: (a) N=1, (b) the published protocol N=3 with the model's argmax path as iteration-2 input, and (c) a diagnostic N=3 where iteration-2 input is the ground-truth CTC oracle path. Repeat across at least 5 random training seeds if checkpoints are available; otherwise bootstrap over test PSMs to obtain 95% confidence intervals for AA precision and peptide recall. If (c) clearly beats (a) while (b) does not, or if the (b)-gain over (a) falls inside the confidence interval, the oracle-to-argmax transfer assumed by Section 3.4 is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Algorithm 1 (Section 3.3) and the Appendix A code, the conditional input fed to the decoder is always derived from ground-truth target tokens: the best CTC alignment is computed from the model's output probabilities, then oracle tokens are gathered from tgt tokens and blanks are inserted. The decoder is thus trained to denoise an oracle path. In Section 3.4, Eq. 5, inference iterates by feeding y^(i-1) = argmax P(·|I, EmbeddingLayer(y^(i-2))), i.e., the model's own decoded path. That path is a different input distribution—systematic model errors, not ground-truth-derived masks—so the embedding layer is evaluated out-of-distribution whenever argmax deviates from the oracle alignment. If that deviation is large, refinement may lock in errors. The only supporting evidence is Table 5: AA precision 0.848 at iteration 1 to 0.854 at iteration 3, peptide recall 0.728 to 0.736, with no seeds, error bars, or comparison to a same-checkpoint no-refinement baseline; the full ablation (Table 4) shows just +0.012 AA precision over PrimeNovo. Figure 4's note that 'validation does not involve looking at true tokens' implicitly acknowledges the mismatch. Until oracle-conditioned and self-conditioned refinement are compared, the core novelty's benefit is unproven.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes RefineNovo, a non-autoregressive Transformer (NAT) for de novo peptide sequencing built on the authors' earlier PrimeNovo backbone. The two claimed contributions are (i) a CTC-path-based curriculum learning strategy with adaptive difficulty annealing, and (ii) an iterative self-refinement module used at inference. The model is trained on MassIVE-KB and evaluated on the 9-species-V1 and 9-species-V2 benchmarks, reporting average amino acid precision of 0.800 and 0.907 and peptide recall of 0.653 and 0.790, respectively, and claiming to outperform all previous methods. The paper also reports a more than 90% reduction in NAT training failures based on 20 randomly selected training subsets.","tokens_in":21881,"tokens_out":8552,"duration_ms":83793,"significance":"If the reported numbers hold, RefineNovo would set a new state of the art for NAT-based de novo peptide sequencing, and the curriculum strategy would be a practically valuable contribution to stabilizing CTC training in parallel sequence models. The paper has real strengths: it evaluates against external public benchmarks and multiple baseline families, provides an ablation of the three proposed components, reports a training-stability case study, and releases code and models. Those strengths are offset by the fact that the headline gains over PrimeNovo are small (0.012 AA precision on V1, 0.016 on V2), every reported number is a single point estimate with no error bars, and the iterative refinement module is used at inference under a different input distribution from the one it was trained on. The central empirical claim is therefore plausible but not yet established at the standard expected for a benchmark-leading result.","major_comments":[{"comment":"The curriculum schedule is internally inconsistent and cannot be reproduced as written. In Algorithm 1, the variable named 'acc' is computed as acc = 1 − same_num/seq_lens, which is an error rate, and then ρ_ratio is set to α(1−acc), so ρ_ratio equals α times the token matching rate; that gives low masking early and higher masking later, consistent with the 'easy-to-hard' prose description. The prose, however, defines acc as the model's prediction accuracy and states ρ_ratio = α(1−acc), which would give a hard-to-easy schedule. The Appendix A code further complicates matters: it sets keep_prob = ((seq_lens − same_num)/seq_lens)·peek_factor and masks with probability 1−keep_prob. This equals α·matching_rate only when α=1; for any other α, the code's mask probability is 1−α(1−matching_rate), not α·matching_rate. The value of α is never reported. Please resolve the naming conflict, state the intended mask probability explicitly, and report the value of α used in the experiments.","section":"3.3, Algorithm 1, Appendix A"},{"comment":"The iterative refinement module trains on oracle CTC paths constructed from ground-truth target tokens (Algorithm 1 lines 7–9 and Appendix A: oracle = tgt_tokens.gather(-1, oracle_pos)), while at inference Eq. (5) feeds the model's own argmax path y^(i−1) into the same embedding layer. The decoder is therefore never trained on model-generated paths, so the self-refinement procedure is an out-of-distribution use of the embedding layer whenever the argmax path deviates from the oracle alignment. Table 5 reports only the iteration curve (AA precision 0.848 to 0.854, peptide recall 0.728 to 0.736) with no error bars and no comparison to a same-checkpoint single-pass baseline; the ablation in Table 4 attributes only +0.009 AA precision to the refinement component. A controlled comparison is needed between refinement from (a) the oracle path, (b) the model's argmax path, and (c) no refinement, with multiple seeds and with the agreement rate between argmax paths and oracle alignments reported.","section":"3.3, 3.4, Table 5"},{"comment":"All benchmark results are single point estimates with no standard deviations, seeds, or significance tests. The headline differences over PrimeNovo are small: 0.800 vs 0.788 AA precision and 0.653 vs 0.638 peptide recall on V1, and 0.907 vs 0.891 AA precision and 0.790 vs 0.777 peptide recall on V2. The ablation differences are of the same small magnitude. Without variance estimates, per-species paired tests, or at least multiple runs, the abstract's claim that RefineNovo 'outperforms all previous methods' is not statistically supported. Please report mean and standard deviation over at least three seeds and, where possible, per-species paired comparisons against PrimeNovo.","section":"Tables 1, 2, 4, 5"},{"comment":"The claim of a more than 90% reduction in training failures rests on 20 random subsets from MassIVE-KB, with PrimeNovo failing in 18 runs and RefineNovo failing once. The manuscript does not define the failure criterion, describe how the subsets were drawn, or report the random seeds; moreover, Figure 4 states that the case-study plots come from different datasets, so the displayed curves are not a same-data comparison. This claim should be made reproducible by specifying the failure criterion (e.g., loss NaN, validation collapse, divergence threshold) and by reporting the same-dataset comparisons for both models.","section":"4.2, Training Success Rate and Case Study"}],"minor_comments":[{"comment":"The sinusoidal encoding formula is malformed; the expression as printed is missing operators and parentheses and cannot be parsed. Please rewrite it in a standard form.","section":"3.2, Eq. (1)"},{"comment":"Equation (3) writes L = P(A|ρ(A,y),I), but the quantity being optimized should be a loss, i.e., the negative log probability of the target under the conditional input, not the probability itself.","section":"3.3, Eq. (3)"},{"comment":"The caption of Table 5 says 'Effect of different beam sizes,' but the rows report the number of iterative refinement steps. The caption should be corrected.","section":"Appendix B, Table 5"},{"comment":"The main text says candidate CTC paths are re-ranked during training to select the oracle path, while Figure 3's inference phase mentions 'Argmax CTC Path Sampling' without describing whether reranking is used at inference. Please clarify the inference-time path selection procedure.","section":"3.3 and Figure 3"},{"comment":"The text states an initial learning rate of 4e-4 that is 'gradually increased to the target peak within the first epoch,' which is ambiguous about whether the initial value or the peak is 4e-4; please specify the warm-up schedule.","section":"4.1, Model Details"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from the same group as PrimeNovo, and the comparison against PrimeNovo in Tables 1 and 2 appears to use numbers reported in the original PrimeNovo publication rather than a fresh head-to-head retraining under identical conditions. Given how small the reported gains are, I would ask the editor to consider requesting such a direct comparison. The core mechanism of the paper, the curriculum and refinement modules, is not in question as an idea, but the evidence as presented is not yet sufficient for the strong SOTA claim. The issues identified are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"RefineNovo is a plausible incremental improvement over PrimeNovo, not a breakthrough. The benchmark numbers are consistent and the training-stability result is striking, but the strongest claims currently outrun the evidence.\n\nWhat is actually new is narrow: glancing-mask curriculum learning adapted to CTC path sampling for NAT peptide sequencing, with a performance-annealed mask ratio, plus an iterative refinement module that feeds decoded tokens back through the learned embedding layer. The ablation shows each component helps, and the gains on 9-species-V1/V2—roughly +0.012 AA precision on V1 and +0.016 on V2—are modest but consistent across most species. The NovoBench evaluation is a useful addition. If the 90% reduction in training failures holds up, that is the most practically valuable result in the paper.\n\nThe soft spots are real. Algorithm 1 is genuinely confusing: the variable called \"acc\" is defined as 1 minus the matching rate, and then the masking ratio is computed as alpha*(1-acc), which makes the direction of the curriculum ambiguous. The appendix code computes a \"keep probability\" instead, and the three formulations don't obviously agree. The authors should rewrite this section with one consistent equation.\n\nThe headline claim \"outperforms all previous methods\" is not supported because ContraNovo—the strongest autoregressive baseline by their own description—is absent from Tables 1 and 2, and the text admits ContraNovo is better on human data. That overstatement is easy to fix but needs fixing. There are also no error bars or repeated seeds anywhere; with gains of 0.006–0.016, those small deltas could easily be noise.\n\nThe train/inference mismatch raised in the stress-test note is a fair question, but Table 5 partially answers it: iteration 1 (no refinement) at 0.848 vs iteration 3 at 0.854 shows self-conditioned refinement does help on the model's own argmax path. The improvement is small and still lacks error bars, so a formal comparison between oracle-conditioned and self-conditioned refinement would settle it. The training-stability experiment also lacks a defined failure criterion; \"loss explosion and extreme overfitting\" need operational thresholds.\n\nThe code link is promised but absent, which is a problem for a paper whose value is empirical. Overall, the method is coherent and the results are likely in the right ballpark. This deserves a serious referee, but the claims need to be scaled to the evidence and the missing reproducibility items supplied. If I were working in de novo sequencing, I'd want to see the code and error bars before citing it.","headline":"RefineNovo is a plausible incremental improvement over PrimeNovo, but the state-of-the-art claim outruns the evidence: a key baseline is missing, there are no error bars, and the masking schedule is ambiguous.","tokens_in":22380,"tokens_out":4351,"would_cite":false,"duration_ms":44299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RefineNovo claims a curriculum-plus-refinement scheme makes non-autoregressive peptide sequencing the best on two nine-species benchmarks.","keywords":["de novo peptide sequencing","non-autoregressive transformer","curriculum learning","CTC loss","iterative refinement","mass spectrometry","difficulty annealing","protein sequence prediction"],"falsifier":"Run RefineNovo at inference with the first pass replaced by a random or deliberately corrupted CTC path and measure whether subsequent refinement degrades; also train a control that conditions on the model's own argmax path during training instead of the oracle path. If the control matches or beats RefineNovo, the oracle-path curriculum is not the source of the improvement, and if corrupted conditioning propagates errors, the refinement transfer assumption is false.","tokens_in":21359,"feed_emoji":"🧬","tokens_out":5569,"duration_ms":51416,"temperature":0.7,"pith_summary":"The paper argues that the main obstacle to non-autoregressive (NAT) peptide sequencing is not the architecture but the CTC training objective, which is unstable and frequently fails to converge. It introduces RefineNovo, which combines a structured protein-sequence curriculum with an inference-time self-refinement loop. The curriculum samples the model's own most probable CTC alignment of the true peptide, reveals those tokens through a dynamically annealed mask, and thereby eases the model from nearly supervised prediction to full de novo decoding. On the 9-species-V1 and 9-species-V2 benchmarks the paper reports average amino acid precision of 0.800 and 0.907 and peptide recall of 0.653 and 0.790, surpassing all previous methods, and a reduction of NAT training failures by more than 90 percent.","feed_headline":"RefineNovo beats all prior peptide sequencers on nine species","feed_subtitle":"A CTC-path curriculum and self-refinement cut NAT training failures by 90% and lift amino acid precision to 0.907.","key_machinery":"The load-bearing object is the CTC oracle-path curriculum mask. Given the target peptide $A$ and a CTC path $y'$ sampled as the most probable valid alignment of $A$, the model computes an oracle token at each position by taking the label at the midpoint of $y'$ and blanking where the path parity requires; the fraction of positions masked is $\\rho=\\alpha(1-\\text{acc}(A,y_{\\text{argmax}}))$, so early training shows many true tokens and later training masks more. This converts the NAT objective from independent prediction $P(A|I)$ into conditioned prediction $P(A|\\rho(A,y'),I)$, shrinking the effective search space, and it is what makes the embedding layer able to consume arbitrary CTC paths. At inference the same embedding layer is fed the argmax path repeatedly, and the precise-mass-control knapsack decoder enforces the precursor-mass constraint.","core_discovery":"The central claim is that a CTC-trained NAT decoder becomes both more stable to train and more accurate at inference if it is conditioned, during training, on a partially revealed oracle CTC path rather than on an empty decoder input. The method computes the most probable valid CTC alignment of the ground-truth peptide, extracts the amino-acid tokens at the midpoint of that alignment, blanks out positions according to a mask ratio that anneals with the model's own argmax accuracy, and feeds the resulting masked sequence into the embedding layer. The same embedding layer is then reused at inference: the argmax-decoded CTC path from the previous pass is embedded and fed back, yielding $y^{(i)}=\\arg\\max P(\\cdot|I,\\text{EmbeddingLayer}(y^{(i-1)}))$, repeated a few times before the precise-mass-control dynamic program decodes the final sequence. The paper reports that this combination achieves state-of-the-art results on both benchmark versions across most of the nine species, with ablation experiments attributing the gain to the interaction of curriculum masking, difficulty annealing, and iterative refinement.","pith_inferences":["The refinement module actually recycles the CTC path rather than the reduced peptide, which suggests the method is learning to search alignment space; a direct comparison against masked-token diffusion refinement of the peptide sequence would isolate whether alignment-level recycling is the mechanism.","Because the curriculum uses the model's own argmax accuracy as the difficulty meter, the same recipe should transfer to other CTC-trained NAT tasks such as speech recognition or handwriting recognition, where loss instability is also reported.","The NovoBench domain-shift result (0.09 peptide score on 7-species yeast for both PrimeNovo and RefineNovo) implies the gains concentrate in the training distribution; a testable extension is to retrain on mixed-instrument data and check whether the oracle-path curriculum closes the distribution gap.","The embedding layer trained on oracle CTC paths accepts arbitrary pseudo-labels, so one could feed unlabeled spectra decoded by the model itself and fine-tune semi-supervised, using the curriculum mask as a denoising objective."],"forward_implications":["On the 9-species-V1 benchmark, RefineNovo leads in amino acid precision and peptide recall for eight of nine species (averages 0.800 and 0.653), making it the direct reference point for future de novo sequencing work.","On the stricter 9-species-V2 benchmark the model reports the best amino acid precision for all nine species and peptide recall for eight, with averages 0.907 and 0.790.","The curriculum reduces NAT training failures by more than 90 percent: in 20 sampled MassIVE-KB training runs, PrimeNovo failed 18 times while RefineNovo failed once.","Iterative refinement saturates quickly: peptide recall rises from 0.728 at one pass to 0.736 at three passes and then plateaus, so the accuracy gain can be obtained at three decoding iterations.","The ablation shows fixed masking at ratio 0.7 hurts (0.733 amino acid precision) whereas dynamic annealing plus curriculum plus refinement reaches 0.800, so the adaptive difficulty schedule is doing real work."],"supporting_citations":[{"why":"supplies the PrimeNovo NAT backbone, the CTC-based training setup, and the Precise Mass Control decoder that RefineNovo extends and must outperform.","marker":"(Zhang et al., 2025)"},{"why":"introduces non-autoregressive generation and identifies the multi-modal optimization problem that motivates replacing cross-entropy with CTC.","marker":"(Gu et al., 2017)"},{"why":"defines the CTC loss and its alignment reduction rules that the curriculum masking operates on.","marker":"(Graves et al., 2006)"},{"why":"provides DeepNovo, the first deep-learning baseline, and the original 9-species-V1 benchmark test set.","marker":"(Tran et al., 2017)"},{"why":"contributes CasanovoV2 and the revised 9-species-V2 benchmark; its performance is the main autoregressive baseline.","marker":"(Yilmaz et al., 2024)"},{"why":"supplies the glancing-masking strategy whose oracle-based masking idea the difficulty-annealed curriculum adapts to CTC paths.","marker":"(Qian et al., 2020)"},{"why":"represents the PEAKS database-search/DB baseline in all benchmark tables.","marker":"(Ma et al., 2003)"},{"why":"provides MassIVE-KB, the training corpus used for all model training and for the 20-split training-stability experiment.","marker":"(Wang et al., 2018)"}],"fun_headline_variants":["Curriculum learning cuts NAT training failures by 90%","CTC mask curriculum improves peptide sequencing on 9 species","Self-refinement makes NAT peptide sequencer state-of-the-art","NAT peptide sequencing stabilized via curriculum and self-refinement","Sampled CTC curriculum boosts peptide sequencing accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method trains the decoder on partially revealed oracle CTC paths, but at inference it feeds back the model's own greedy argmax path; the whole accuracy gain rests on the assumption that the model learns to condition on arbitrary CTC paths as well as it conditions on correct ones.","fun_headline_variants_meta":{"raw":{"variants":["Curriculum learning cuts NAT training failures by 90%","CTC mask curriculum improves peptide sequencing on 9 species","Self-refinement makes NAT peptide sequencer state-of-the-art","NAT peptide sequencing stabilized via curriculum and self-refinement","Sampled CTC curriculum boosts peptide sequencing accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1540,"prompt_tokens":984,"completion_tokens":556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":600,"tokens_out":556,"duration_ms":5959,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:00:20.554139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RefineNovo at inference with the first pass replaced by a random or deliberately corrupted CTC path and measure whether subsequent refinement degrades; also train a control that conditions on the model's own argmax path during training instead of the oracle path. If the control matches or beats RefineNovo, the oracle-path curriculum is not the source of the improvement, and if corrupted conditioning propagates errors, the refinement transfer assumption is false.","supporting_citations":[{"cited_title":"Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks","cited_arxiv_id":null,"evidence_quote":"defines the CTC loss and its alignment reduction rules that the curriculum masking operates on."}],"review_version":2}