{"id":"d67a2415-2bbe-4a4d-8931-5744bf66f10d","arxiv_id":"2412.05569","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SMI-Editor pre-trains a SMILES Transformer to restore randomly dropped chemical fragments via Levenshtein edit operations, improving downstream molecular property prediction.","lead":"SMI-Editor is a SMILES language model pre-trained with an edit-based objective: it drops random chemical fragments from a molecule and trains the model to restore them through deletion and insertion operations. The resulting representation beats several existing SMILES and graph models on molecular property prediction, though the benchmark comparisons need a contamination check.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an unchecked assumption that the 19M-molecule pre-training corpus does not overlap the MoleculeNet test sets; the appendix's SMI-MLM BBBP=89.4 suggests leakage and would differentially favor SMI-EDITOR, so a contamination check is required.","rationale":"The reader and I converge on the same weakest point: the paper never establishes that the 19M pre-training corpus is disjoint from the MoleculeNet test splits. I agree this is the load-bearing assumption for the headline claims. The mechanism makes the concern concrete: SMI-EDITOR's pre-training directly optimizes whole-molecule reconstruction through deletion, placeholder insertion, and token prediction, so a duplicated test molecule is memorized as a full string. SMI-MLM's token-masking objective sees the same molecule only as 15%-masked fragments, so it has far less capacity to memorize the exact test item. Hence overlap would not merely shift both models equally; it could differentially inflate the very gap the paper attributes to fragment-level supervision. The Appendix J SMI-MLM BBBP number is an additional warning sign, though it is not by itself proof of leakage. I would not move the verdict to REJECT: the method, ablations, and matched SMI-MLM comparisons are coherent, and the overlap check could come back clean. I would keep the reader's CONDITIONAL verdict, explicitly conditioned on a contamination analysis and, ideally, seed-averaged error bars on the exact split. This is why I set verdict_should_be to UNCHANGED.","tokens_in":26891,"tokens_out":5965,"duration_ms":57325,"concrete_test":"Download the Zhou et al. (2023) 19M pre-training SMILES; canonicalize both it and the MoleculeNet train/valid/test molecules with RDKit using exact canonical SMILES and InChI keys; then intersect the pre-training set with each of the seven test splits and report overlap counts and percentages. After that, re-run the Section 4 fine-tuning protocol for SMI-EDITOR and SMI-MLM after excluding from the test and validation sets any molecule present in pre-training. If the SMI-EDITOR minus SMI-MLM mean ROC-AUC gap in Table 1 shrinks by more than roughly one point, or if SMI-EDITOR no longer beats the best baselines on a majority of datasets, the SOTA and fragment-supervision claims should be revised downward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of Table 1 is that SMI-EDITOR's mean ROC-AUC (77.8) exceeds SMI-MLM (73.2) and published baselines because fragment-level edit pre-training transfers substructure knowledge. This conclusion depends on an unstated assumption: the 19M-molecule pre-training corpus from Zhou et al. (2023), described in Section 4.1, does not contain the molecules in the MoleculeNet test splits. The paper never checks or reports this overlap. The assumption is load-bearing because the pre-training objective is to regenerate whole molecules from dropped fragments; if a test molecule appears in the 19M set, SMI-EDITOR has been explicitly trained to reconstruct that exact SMILES, whereas SMI-MLM's 15%-token masking objective provides much weaker whole-molecule memorization. Contamination would therefore inflate the SMI-EDITOR-vs-SMI-MLM margin and all headline SOTA comparisons. The appendix already contains a warning sign: Table 8 reports SMI-MLM on DeepChem splits at 89.4 for BBBP, far above every published scaffold-split BBBP baseline listed in the same table, while on the paper's own scaffold split SMI-MLM scores only 68.6 (Table 1). Such an outlier is the expected fingerprint of leakage, although a split/protocol mismatch could also explain it. Until the overlap is quantified and results are recomputed on a withheld, non-overlapping test set, the causal attribution to fragment-level supervision is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMI-EDITOR, a SMILES language model pretrained with an edit-based objective: random molecular fragments are dropped, and the model learns to restore the original SMILES via token deletion, placeholder insertion, and token prediction, using Levenshtein-distance expert actions (LevT) and a dual deletion loss. The authors argue that MLM pretraining on SMILES suffers from rapid saturation and fails to capture substructure semantics, and they provide analyses supporting that SMI-EDITOR mitigates these issues. Experiments on 7 MoleculeNet classification tasks report a mean ROC-AUC of 77.8 for SMI-EDITOR, exceeding the matched MLM baseline SMI-MLM (73.2) and several 2D/3D baselines. Ablations confirm the contributions of fragment-level dropping and the three editing operations.","tokens_in":27138,"tokens_out":10455,"duration_ms":86857,"significance":"If the results hold, SMI-EDITOR is a conceptually valuable contribution: it is, to my knowledge, the first edit-based pretraining framework for SMILES, and it demonstrates a clear and consistent improvement over a matched MLM baseline. The paper includes open-source code, ablation studies, and novel analytical experiments on substructure semantics, which are useful for the community. However, the central empirical claim requires stronger validation due to potential data contamination and missing statistical rigor.","major_comments":[{"comment":"The central SOTA claim depends on an unchecked assumption that the 19M-molecule pretraining set from Zhou et al. (2023) does not overlap the MoleculeNet test splits. The paper never reports an overlap analysis. The appendix's DeepChem-split result for SMI-MLM (BBBP 89.4, Table 8) is far above published baselines (e.g., ChemBerta 64.3, MolCLR 73.6), which is the expected fingerprint of leakage. Because SMI-EDITOR is trained to reconstruct whole molecules from dropped fragments, contamination would inflate its scores more than SMI-MLM's token-level masking. Please report the exact overlap (e.g., canonical SMILES or InChI matching) between the pretraining corpus and every test split, and rerun the main comparison on a decontaminated subset or after removing overlapping molecules.","section":"Section 4.1, Table 1, Appendix J"},{"comment":"The headline results are reported as single-run ROC-AUC values without error bars or significance tests. The mean gap over SMI-MLM (77.8 vs 73.2) is large, but per-task differences are more modest (e.g., SIDER 63.0 vs 59.6, ToxCast 67.4 vs 69.2 for MolCLR), and run-to-run variance in fine-tuning can exceed one point. Please report means and standard deviations over at least five fine-tuning seeds for SMI-EDITOR and SMI-MLM on all seven tasks, and indicate whether the differences are statistically significant.","section":"Table 1"},{"comment":"Uni-Mol (Zhou et al., 2023) is the source of the pretraining dataset and the fine-tuning split used in this paper, yet it is omitted from Table 1. Without Uni-Mol, the claim of 'outperforming several 3D molecular representation models' omits the most directly relevant 3D baseline. Please add Uni-Mol's results for the same splits, or state explicitly if its published numbers were obtained under a different protocol and are therefore not directly comparable.","section":"Section 4.2, Table 1"},{"comment":"The main text states that pretraining uses the 19M-molecule dataset from Zhou et al. (2023), but Appendix J says the DeepChem-split experiments use SMI-EDITOR 'pre-trained on datasets provided by Ross et al. (2022).' This inconsistency makes Table 8 difficult to interpret. Please clarify which pretraining checkpoint is used for each experiment, and confirm whether SMI-MLM in Table 8 was trained on exactly the same pretraining data and protocol as SMI-EDITOR.","section":"Appendix J vs Section 4.1"}],"minor_comments":[{"comment":"The phrase 'and even outperforming several 3D molecular representation models' should be 'and even outperforms several 3D molecular representation models'.","section":"Abstract"},{"comment":"The sentence 'These findings highlight the limited scalability of SMILE MLMs' contains a typo: 'SMILE' should be 'SMILES'.","section":"Section 2.1"},{"comment":"The text refers to the 'unrealistic special symbol [MASA]' but the intended symbol is '[MASK]'.","section":"Section 4.3.1"},{"comment":"The text says SMI-EDITOR achieves SOTA performance on 4 out of 7 tasks, but the table shows it is best on 5 tasks (BBBP, Tox21, SIDER, MUV, ClinTox); please correct this count.","section":"Table 1"},{"comment":"The display equation for the dual deletion loss is typeset in a confusing way (Σ_{y_i ∈ M} d*_i ∈ d* ...); please introduce the notation for d*_i and the summation index more clearly.","section":"Section 3.2, Eq. (4)"},{"comment":"The column header 'Max Tokens' likely refers to maximum tokens per batch rather than a model dimension; please rename it to avoid ambiguity.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an ICLR 2025 paper and has clearly received some revision, but the current version still lacks a decontamination check between the pretraining corpus and the MoleculeNet test sets. This is the single most important risk to the central claim, and the editor should require the authors to provide this analysis before publication. The missing Uni-Mol baseline is also surprising given that the pretraining data and splits are taken from that work; I recommend the editors treat this as a required addition rather than optional."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe interesting part of this paper is the pretraining idea: take a valid SMILES, drop whole BRICS/RMCF fragments, and train a transformer to reconstruct the original via deletion/insertion edits (LevT-style). That is new for SMILES, and the saturation analysis for MLMs is a nice, useful observation. The ablations are honest and mostly support the design: fragment dropping beats atom dropping, and every editing loss contributes. The appendices even report a failed variant (SMI-EDITOR-Cor) and a cost comparison—rare transparency.\n\nThat said, the headline 'state-of-the-art' claim rests on an assumption the paper never checks: that the 19M-molecule pretraining set (from Zhou et al.) does not overlap with the MoleculeNet test splits. Since the objective is to reconstruct whole molecules from dropped fragments, any test molecule seen in pretraining would be memorized and inflate the reported ROC-AUC. The appendix already contains a red flag: SMI-MLM on DeepChem splits scores 89.4 on BBBP, far above every published baseline listed in the same table, despite the same pretraining set and a plain MLM objective. That is exactly the fingerprint of leakage, or at least a split/protocol mismatch that needs explanation. The main Table 1 has no error bars, so the gap between SMI-EDITOR and SMI-MLM might partly be noise.\n\nThe other comparisons are a bit overstated too: claiming to beat 3D models is not controlled, because those baselines use different pretraining data and protocols. But the core methodological claim—that fragment-level edit supervision gives better representations than token masking—is plausible, and the ablations support it.\n\nBottom line: this is a serious paper with a load-bearing missing check. If it came to me for review, I would send it to referees but ask for a contamination analysis (overlap between pretraining and test sets), recomputed results on guaranteed-disjoint splits, and error bars. Worth reading for anyone working on SMILES pretraining; just don't cite the SOTA numbers until the leakage question is resolved.","headline":"Genuinely new edit-based SMILES pretraining objective and a useful MLM saturation analysis, but the SOTA claim needs a contamination check before the numbers are trustworthy.","tokens_in":27764,"tokens_out":2699,"would_cite":false,"duration_ms":24306,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SMI-Editor, a SMILES language model pre-trained with edit operations over molecular fragments, outperforms token-masked baselines on seven MoleculeNet classification tasks, reaching an average ROC-AUC of 77.8.","keywords":["SMILES language model","molecular property prediction","edit-based pretraining","fragment-level supervision","Levenshtein transformer","masked language modeling","MoleculeNet","substructure semantics"],"falsifier":"Compute the exact overlap between the SMILES in the pretraining corpus and the molecules in each MoleculeNet test set after canonicalization and salt stripping; if any non-trivial fraction of test molecules appears in pretraining, retrain SMI-MLM and SMI-Editor on the deduplicated corpus and compare the gap. A large drop in SMI-Editor's advantage would indicate the edit objective's gains were partly memorization; a sustained gap would confirm transferable fragment learning.","tokens_in":26576,"feed_emoji":"🧪","tokens_out":5242,"duration_ms":42643,"temperature":0.7,"pith_summary":"This paper argues that masked-token pretraining is the wrong supervision for SMILES language models: molecules are understood through substructures, and token-level masking saturates quickly and fails to capture functional-group semantics. The authors propose SMI-Editor, which deletes random molecular fragments from valid SMILES strings and trains the model to restore the original SMILES through deletion and insertion edits. On seven MoleculeNet classification tasks, SMI-Editor reaches an average ROC-AUC of 77.8, ahead of 73.2 for a matched MLM baseline and ahead of several 3D molecular representation models. If the result holds, it means the choice of pretraining objective, not just scale or architecture, is what unlocks molecular semantics from SMILES.","feed_headline":"Edit-based SMILES pretraining beats token-masking on 7 molecule tasks","feed_subtitle":"Fragment-level supervision lifts mean ROC-AUC from 73.2 to 77.8 on MoleculeNet.","key_machinery":"The machinery is the Levenshtein-style editing head on top of a Transformer encoder, combined with fragment-level supervision generated by a modified BRICS/RMCF fragmentation. Given a SMILES string with selected fragments dropped, the model predicts three operations: which tokens to delete, how many placeholders to insert at each position, and which vocabulary token fills each placeholder. The dual deletion loss, which supervises deleting erroneous tokens directly on the corrupted input rather than only on model-generated insertions, is what makes deletion learning non-trivial. Together these operations convert pretraining from a single-token prediction task into a structure reconstruction task over chemically meaningful fragments.","core_discovery":"SMI-Editor's central claim is that an edit-based objective with fragment-level supervision lets a SMILES-only transformer learn molecular substructure semantics that masked language modeling misses. During pretraining, the model is given a valid SMILES string with randomly deleted substructures and must reconstruct the original string by predicting token deletions, placeholder insertions, and tokens for placeholders, with a dual deletion loss that supervises removal of corrupted tokens. The paper shows this objective avoids the rapid saturation that makes MLM pretraining trivial, and that removing any of the editing signals degrades downstream performance. The result is presented as a new state of the art on MoleculeNet property classification, including gains over 3D models.","pith_inferences":["Because the paper never reports whether the 19M pretraining molecules overlap the MoleculeNet test sets, the cleanest extension of this work is to rerun the comparison with a deduplicated corpus; the ROC-AUC gap would shrink if memorization, not fragment knowledge, drove part of the gain.","The edit-based objective resembles a denoising autoencoder over chemical structures; connecting SMI-Editor's fragment perturbations to reaction templates or retrosynthetic disconnections could make the reconstruction task chemically meaningful beyond BRICS fragments.","The paper's analysis of ESOL shows molecular-weight confounds can mimic substructure sensitivity; future evaluations of 'substructure understanding' should control for size before attributing prediction changes to functional-group semantics.","An encoder-only model cannot generate, but the appendix's SMI-Editor-Gen extends the same fragment-reconstruction idea to sequence-to-sequence retrosynthesis; combining the edit objective with contrastive or 3D alignment signals is a natural next step."],"forward_implications":["Fragment-level edit supervision can be added to any SMILES encoder without changing the downstream fine-tuning protocol, so the reported gains should transfer to other encoder-based molecular property models.","The same pretraining objective, adapted to an encoder-decoder, reaches top results on USPTO-50K retrosynthesis in the paper's appendix, suggesting the editing task learns generative chemical knowledge, not just classification features.","Because the edit objective avoids rapid saturation, bigger models keep improving with more capacity and training steps, whereas the paper's matched MLM baseline plateaus at small scale.","The paper's ablation indicates placeholder insertion, the operation that localizes missing fragments, is the largest single contributor, meaning positional reasoning about where substructures belong is a key learned skill.","Under an equal training budget, SMI-Editor still outperforms an MLM trained three times longer, suggesting the benefit comes from the objective rather than from extra computation."],"supporting_citations":[{"why":"Supplies the Levenshtein Transformer editing framework, including deletion, placeholder insertion, and token prediction, which SMI-Editor adapts.","marker":"Gu et al. (2019)"},{"why":"Provides the BRICS fragmentation rules used to split molecules into chemically meaningful fragments.","marker":"Degen et al. (2008)"},{"why":"Provides the RMCF fragmentation extension that further splits ring-side-chain connections, yielding smaller fragments.","marker":"Wang et al. (2022a)"},{"why":"Supplies the 19M-molecule pretraining corpus and the scaffold-split fine-tuning protocol used in the main experiments.","marker":"Zhou et al. (2023)"},{"why":"Defines the MoleculeNet benchmark and datasets used for downstream property prediction evaluation.","marker":"Wu et al. (2018)"},{"why":"Defines the masked language modeling objective that serves as the baseline that SMI-Editor is compared against.","marker":"Devlin et al. (2018)"}],"fun_headline_variants":["SMILES editor beats token masking on molecule benchmarks","Fragment-level SMILES editing raises ROC-AUC to 77.8","Edit-based SMILES pretraining outperforms 3D models","SMI-Editor: teach SMILES models by fixing broken strings","Molecular LM learns from fragment edits, not just masks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that the 19-million-molecule pretraining set does not contain the molecules used in the MoleculeNet test splits, so the reported ROC-AUC gains reflect transferred fragment knowledge rather than memorization of benchmark molecules.","fun_headline_variants_meta":{"raw":{"variants":["SMILES editor beats token masking on molecule benchmarks","Fragment-level SMILES editing raises ROC-AUC to 77.8","Edit-based SMILES pretraining outperforms 3D models","SMI-Editor: teach SMILES models by fixing broken strings","Molecular LM learns from fragment edits, not just masks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1434,"prompt_tokens":923,"completion_tokens":511,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":539,"tokens_out":511,"duration_ms":4660,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:35:11.767032+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the exact overlap between the SMILES in the pretraining corpus and the molecules in each MoleculeNet test set after canonicalization and salt stripping; if any non-trivial fraction of test molecules appears in pretraining, retrain SMI-MLM and SMI-Editor on the deduplicated corpus and compare the gap. A large drop in SMI-Editor's advantage would indicate the edit objective's gains were partly memorization; a sustained gap would confirm transferable fragment learning.","supporting_citations":[],"review_version":1}