{"id":"097d13d5-19da-4e02-8a3b-9dafafb3a533","arxiv_id":"2411.14464","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"JESTR ranks metabolite candidates by cosine similarity in a jointly learned molecule-spectrum embedding space, outperforming ESP, MIST, and CMSSP on benchmark datasets.","lead":"JESTR is a machine learning tool that ranks candidate chemical structures for an unknown mass spectrum by learning a shared embedding space for molecules and spectra. It reports large improvements over existing annotation tools on four metabolomics datasets, which could speed up identifying metabolites in biological samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Regularization candidates and test candidate sets are drawn from the same PubChem formula pools, yet the paper never measures their overlap; if test decoys were seen during training, the reported rank improvements over ESP/MIST could be inflated.","rationale":"I focused on the assumption that most directly underwrites the central claim: that the molecules used as regularization candidates during training are effectively disjoint from the molecules that will appear in test candidate sets. The paper never checks this. Section 2.3 and Section 3.1 use the same formula-based PubChem retrieval for regularization and for test candidate construction, while the splits only guarantee InChIKey-disjointness of target molecules. Because the regularization loss explicitly pushes candidate embeddings away from spectrum embeddings, any test decoy already used as a regularization candidate has been partially pre-ranked against spectra of same-formula molecules. This could inflate the reported rank@k improvements over ESP, MIST, and CMSSP, which did not see those decoys in training. The proposed overlap analysis and clean-subset re-evaluation would decisively resolve the concern. I do not see the omission as evidence of bad faith; it is an evaluation gap that the authors can close with bookkeeping already present in their training pipeline. The reader identified the same weakest assumption, so I agree with the conditional verdict and recommend no change.","tokens_in":16949,"tokens_out":7719,"duration_ms":83472,"concrete_test":"Compute the InChIKey intersection between (a) the set of molecules used as regularization candidates during JESTR training (ideally logged by the released code; otherwise all PubChem candidates for training formulas up to kaug=32 per molecule per epoch) and (b) each test spectrum's candidate set. For each dataset, report the mean and median fraction of test candidates that were seen in training. Then re-evaluate the released JESTR checkpoint on the subset of test spectra whose candidate sets have zero overlap with the regularization set, and compare rank@1, rank@5, and rank@20 on that clean subset against the full-test numbers. If the clean-subset values drop by more than a few percentage points, the headline gains are inflated by candidate overlap; if they are stable, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 3.3) is that JESTR outperforms ESP, MIST, and CMSSP on rank@1/5/20. JESTR's novel ingredient is regularization (Section 2.3, Eq. 6): for each training molecule, the model is trained to lower cosine similarity between the spectrum embedding and embeddings of up to 32 PubChem candidates sharing the target's formula, using roughly 7 million candidates over the last 3% of epochs. The test candidate sets (Section 3.1) are constructed by the same formula-based PubChem retrieval. The train/test splits are structure-disjoint by InChIKey, but that only excludes test target molecules from the training targets; it does not exclude test candidate decoys (or even the test targets themselves) from the regularization pool. If a molecule appearing in a test candidate set was used as a regularization negative, JESTR has already been optimized to push that exact molecule's embedding away from spectra of same-formula molecules. At inference, this can inflate rank@k because those decoys have been pre-suppressed. The paper provides no overlap analysis between the regularization molecules and the test candidate sets, so the 55.5% and 56.6% average improvements over ESP and MIST may partly reflect exposure to test decoys. This concern is especially sharp because the regularization ablation in Section 3.6 is presented as evidence of the method's value.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JESTR, a contrastive-learning method that embeds mass spectra and molecular graphs in a shared space and ranks candidate molecules by cosine similarity, adding a regularization loss that uses PubChem formula-matched candidate molecules to push non-targets away. The method is evaluated on NPLIB1, NIST2020, MoNA, and MassSpecGym, with rank@1/5/20 comparisons against ESP, MIST, CMSSP, and against SIRIUS and CFM-ID on MassSpecGym subsets. The authors report large average improvements over ESP, MIST, and CMSSP, and an ablation showing that regularization improves rank@1 on three datasets.","tokens_in":17202,"tokens_out":7254,"duration_ms":62622,"significance":"The paper's central idea of avoiding explicit fingerprint or spectrum reconstruction is well motivated, and the evaluation is broad, covering four datasets and multiple baselines. The authors are honest about several cases where baselines beat JESTR, such as MIST on MoNA rank@1 and CMSSP pretrained on NPLIB1 rank@1, and they report that regularization does not help on MassSpecGym. The release of code and data is a strength. However, two methodological issues—possible overlap between regularization candidates and test candidate sets, and hyperparameter selection without a validation split—currently make the magnitude of the reported improvements uncertain. If these are resolved, the paper would be a solid contribution to the metabolomics annotation literature.","major_comments":[{"comment":"The regularization candidates and the test candidate sets are both retrieved from PubChem by molecular formula, yet the paper never measures their overlap. The InChIKey-disjoint split only excludes test target molecules from training targets; it does not prevent test candidate decoys or even test target molecules from appearing as regularization negatives. If such overlap is substantial, the regularization loss of Eq. (6) may have directly suppressed test decoys during training, inflating the rank@k results in Table 2 and the ablation gains in Section 3.6. Please quantify, for each dataset, the fraction of test candidate molecules (and test targets) that appear in the regularization pool, and re-run the main experiments with overlapping candidates removed (or with a candidate-disjoint split) to establish that the reported improvements over ESP, MIST, and CMSSP are not an artifact of this exposure.","section":"Section 2.3 and Section 3.1"},{"comment":"Hyperparameters were selected by grid search and 'the values of the parameters that achieved the best performance were selected and used to train and test the model for all datasets.' Because there is no separate validation split, the reported test numbers are selected on the test set, which is circular and can overstate accuracy. Please introduce a validation split (or nested resampling) and report test results using hyperparameters chosen solely from validation data. The same concern applies to how hyperparameters were selected for the MIST and ESP baselines, which is not described.","section":"Section S2, Table S1"},{"comment":"The text states that regularization improves rank@1 by 11.4%, 6.0%, and 37.1% on NPLIB1, NIST2020, and MoNA, respectively, but the abstract and conclusion report an average improvement of 5.72% across all datasets. This number does not correspond to any simple average of the per-dataset improvements in Table 2, especially since regularization is neutral or slightly negative on MassSpecGym. Please clarify the averaging procedure or correct the reported average.","section":"Section 3.6 and Conclusion"}],"minor_comments":[{"comment":"The MassSpecGym dataset is described as containing 32,010 molecules, but later in the same section it is said to contain 29,000 unique molecular structures; these numbers should be reconciled.","section":"Section 3.1"},{"comment":"There is a typo: 'CMF-ID' should be 'CFM-ID' in the sentence reporting a 238.00% improvement.","section":"Section 3.3"},{"comment":"The CFM-ID comparison uses a single random draw of spectra per collision-energy bin; please report results over multiple random draws or provide a seed and variance estimate, since random selection can affect the merged-spectrum evaluation.","section":"Section 3.3"},{"comment":"The header of Table 3 repeats 'CE loss' for two columns; the caption should clarify which columns correspond to the CMSSP architecture and which to the JESTR architecture.","section":"Table 3"},{"comment":"Table S2 reports that JESTR was trained for 800 epochs on NPLIB1, while Section S1 says contrastive learning was stopped at 1000 epochs; please clarify the stopping criterion and the exact number of epochs used.","section":"Section S2"}],"recommendation":"major_revision","confidential_remarks":"The key risk is the overlap between the regularization candidate pool and the test candidate sets. If the authors can show that overlap is small and that results are robust to removing overlapping molecules, the paper would be much stronger. The hyperparameter tuning issue is also fixable but requires re-running experiments with a proper validation split. Both issues are addressable within the scope of a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"JESTR is a serious empirical paper: it embeds molecules and MS/MS spectra in a shared space with contrastive learning, then fine-tunes with a candidate-regularization trick that gives large rank@k gains on most datasets. The evaluation is unusually thorough—four datasets, external baselines, careful filtering for SIRIUS/CFM-ID comparisons—and the ablation honestly reports where regularization does not help, notably on MassSpecGym. The MCES leakage analysis for the pretrained-model comparisons is exactly the right check, and the code is public.\n\nWhat is actually new is the specific configuration: InfoNCE with temperature-scaled cosine similarity plus regularization with formula-matched PubChem candidates as a fine-tuning step. That combination is not in CMSSP or ChemEmbed, and it does beat ESP, MIST, and CMSSP on most metrics. So the paper has a real, if incremental, contribution.\n\nNow the soft spots, in order. First, hyperparameters: the supplement says grid search selected values that \"achieved the best performance\" with no mention of a validation split. If test performance guided the search, the reported numbers are optimistic. This is common but still needs fixing. Second, no error bars or repeated runs; all results are single point estimates. I cannot tell if the gaps between JESTR and MIST are meaningful. Third, and most important: regularization candidates for training and the candidate sets used for evaluation are both sampled from PubChem by formula, but the paper never measures their overlap. A test decoy that was used as a regularization negative during training would already have its embedding pushed away from same-formula spectra, which can inflate rank@k. The InChIKey split excludes test targets from training targets but does not exclude test decoys from the regularization pool. This is a real gap, not a hypothetical one. The authors need to quantify the overlap and ideally rerun with disjoint regularization candidates.\n\nOne more thing: the speculation about CMSSP data leakage is unverified but not central, since they also retrained CMSSP on the same splits and it still underperforms.\n\nBottom line: this deserves peer review and is likely to be useful to the metabolomics community, but it is not ready as-is. I would send it to referees with a request for validation-based hyperparameter selection, repeated runs, and the overlap analysis. Those are doable and would substantially strengthen the claim.","headline":"Solid empirical paper on joint embedding for metabolomics, with a strong evaluation and an honest ablation—but the headline gains over ESP/MIST need an overlap analysis between regularization candidates and test decoys before I trust them.","tokens_in":17791,"tokens_out":3740,"would_cite":true,"duration_ms":37731,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JESTR ranks metabolite candidates by contrastively learning a joint molecule-spectrum embedding space, avoiding explicit fingerprint or spectrum reconstruction.","keywords":["metabolomics annotation","joint embedding space","contrastive learning","mass spectrometry","candidate ranking","molecular fingerprint","regularization","cosine similarity"],"falsifier":"Compute the overlap between the regularization candidate pool (all formula-matched PubChem molecules retrieved for training targets) and the test candidate sets for NPLIB1, NIST2020, MoNA, and MassSpecGym, e.g., by InChIKey; if a large fraction of test candidates appear in the regularization pool, the rank@1 improvements over the no-regularization baseline would be suspect. A complementary falsifier would be retraining with regularization candidates restricted to molecules whose Tanimoto similarity to every test target is low and checking whether rank@1 gains persist.","tokens_in":16703,"feed_emoji":"🧪","tokens_out":9305,"duration_ms":73802,"temperature":0.7,"pith_summary":"JESTR addresses metabolite annotation — assigning a molecular structure to a measured tandem mass spectrum — by treating the molecule and its spectra as two views of the same object and learning a joint embedding space for both. Instead of explicitly predicting a fingerprint or a spectrum and then matching, JESTR trains a graph-neural-network molecular encoder and a spectral encoder with contrastive loss, and ranks candidate molecules by cosine similarity to the query spectrum in that shared space. The paper claims this implicit paradigm outperforms explicit-construction tools (ESP, MIST) and the joint-embedding model CMSSP on rank@1, rank@5, and rank@20 across four datasets, with average rank@[1-20] gains of 55.5%, 56.6%, and 302.56% respectively, and also beats pretrained SIRIUS and CFM-ID on held-out MassSpecGym subsets by 31% and 238%. A fine-tuning regularization step that pulls formula-matched candidate molecules away from training spectra further improves rank@1 on the three non-MassSpecGym datasets. If the claims hold, the field's heavy reliance on reconstructing intermediate fingerprints or spectra may be unnecessary for accurate ranking.","feed_headline":"Joint embedding beats explicit reconstruction for metabolite IDs","feed_subtitle":"Contrastive embeddings lift metabolite rank@k on four datasets, beating SIRIUS and CFM-ID on unseen molecules.","key_machinery":"The central object is the joint molecule-spectrum embedding space learned by two encoders: a graph-convolutional network over molecular graphs and a three-layer MLP over 1000-dimension binned spectra, trained with Contrastive Multiview Coding (CMC) — a contrastive objective using temperature-scaled cosine similarity (InfoNCE loss) that pulls matching views together and pushes non-matching views apart. The second mechanism is regularization: during the last 3% of training, an additional loss minimizes cosine similarity between each spectrum embedding and the $k_{aug}$ most Tanimoto-similar formula-matched candidate molecules fetched from PubChem, teaching the model to suppress near-target candidates. Ranking at inference is then pure cosine similarity between the query spectrum embedding and each candidate molecular embedding, which the paper argues is sufficient because the joint space already separates targets from candidates.","core_discovery":"The paper's central claim is that a molecule and its mass spectrum are views of the same object, so the annotation problem reduces to learning a joint embedding space in which matching molecule-spectrum pairs are close and non-matching pairs are far apart. Using Contrastive Multiview Coding with a temperature-scaled InfoNCE loss, JESTR trains a GNN encoder for molecular graphs and an MLP encoder for binned, intensity-normalized spectra. At inference, candidates are ranked purely by cosine similarity between the spectrum embedding and each candidate's molecular embedding, with no learned ranking layer and no intermediate fingerprint or spectrum reconstruction. The paper reports that this simple scheme outperforms explicit-construction baselines on all four datasets (NPLIB1, NIST2020, MoNA, MassSpecGym) on nearly all ranks, and demonstrates via ablation that the regularization loss, which pushes the most Tanimoto-similar formula-matched candidates away from each training spectrum, is responsible for substantial rank@1 gains on the three curated datasets. The paper also attributes the advantage over CMSSP to the combination of the InfoNCE loss and cosine-similarity ranking, arguing that magnitude normalization prevents embedding-scale artifacts.","pith_inferences":["The paper leaves implicit that the regularization pool (millions of formula-matched PubChem molecules) may overlap the test candidate sets; if a large fraction of test candidates appeared in training regularization, part of the reported rank@1 gains could come from suppressing already-seen identities rather than learning view-invariant chemistry.","The ablation shows regularization lifts rank@1 on the three curated datasets but slightly reduces rank@20 on NPLIB1 and yields no consistent gain on MassSpecGym, hinting that the last-3%-of-epochs, 0.1-weight schedule is dataset-dependent rather than a universal recipe.","Because JESTR's encoders are independent of how candidates are produced, the same joint space could be reused for de novo generation, e.g., by decoding the molecular embedding manifold, connecting this ranking technique to spec-to-mol generation.","A stricter head-to-head against CMSSP's released pretrained weights, rather than weights retrained on each dataset, would clarify how much of the 302.56% margin is architectural (InfoNCE plus cosine ranking) versus a training-from-scratch disadvantage."],"forward_implications":["If JESTR's results hold, explicit reconstruction of fingerprints or spectra is not a necessary step for accurate candidate ranking; a contrastively learned joint space plus cosine similarity is sufficient.","The regularization recipe — fine-tuning with formula-matched candidates drawn from PubChem — sharpens the target-versus-candidate boundary, improving rank@1 by up to 37.1% on the MoNA dataset.","On held-out MassSpecGym subsets, JESTR generalizes to structurally novel molecules better than pretrained SIRIUS (31% relative rank@1 gain) and CFM-ID (238% relative gain), suggesting implicit embedding methods scale better to unseen chemistry.","The temperature-scaled cosine similarity combined with InfoNCE loss, rather than dot-product cross-entropy, is the component that gives JESTR its edge over the similarly structured CMSSP, per the paper's ablation.","Because JESTR needs no spectral library coverage, its annotation capability is not limited by reference spectra availability, directly addressing the low 2.3% library-search identification rate cited in the introduction."],"supporting_citations":[{"why":"Supplies the CMC contrastive multiview coding method and InfoNCE loss that JESTR adopts for its joint embedding training.","marker":"Tian et al. (2020)"},{"why":"ESP, the mol-to-spec baseline whose ensemble spectral prediction JESTR competes against on all four datasets.","marker":"Li et al. (2024)"},{"why":"MIST, the spec-to-FP baseline, and also the source of the NPLIB1 data split used for one evaluation dataset.","marker":"Goldman et al. (2023)"},{"why":"CMSSP, the joint-embedding contrastive baseline whose dot-product cross-entropy design JESTR ablates against.","marker":"Chen et al. (2024)"},{"why":"SIRIUS, the pretrained fingerprint tool JESTR is compared to on held-out MassSpecGym subsets.","marker":"Dührkop et al. (2019)"},{"why":"CFM-ID, the pretrained spectral-prediction tool JESTR is compared to on held-out MassSpecGym subsets.","marker":"Wang et al. (2021)"},{"why":"MassSpecGym, the benchmark dataset that supplies the test split and candidate sets for the held-out comparisons.","marker":"Bushuiev et al. (2024b)"},{"why":"PubChem, the molecular database from which regularization candidates and test candidate sets are retrieved by formula matching.","marker":"Kim et al. (2019)"}],"fun_headline_variants":["Joint embedding outranks explicit reconstruction for metabolite IDs","No fingerprint reconstruction: JESTR embeds molecules and spectra","JESTR beats SIRIUS, CFM-ID by joint embedding for metabolite annotation","Molecule and spectrum as one view: JESTR boosts annotation rank"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the millions of PubChem molecules used as regularization candidates during training do not substantially overlap the molecules in the test candidate sets; if they do overlap, the model could learn to suppress those specific identities and inflate the reported rank@k numbers.","fun_headline_variants_meta":{"raw":{"variants":["Joint embedding outranks explicit reconstruction for metabolite IDs","No fingerprint reconstruction: JESTR embeds molecules and spectra","JESTR beats SIRIUS, CFM-ID by joint embedding for metabolite annotation","Molecule and spectrum as one view: JESTR boosts annotation rank"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1378,"prompt_tokens":1050,"completion_tokens":328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":251}},"tokens_in":666,"tokens_out":328,"duration_ms":3887,"temperature":1.0,"reasoning_tokens":251,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:45:00.993826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between the regularization candidate pool (all formula-matched PubChem molecules retrieved for training targets) and the test candidate sets for NPLIB1, NIST2020, MoNA, and MassSpecGym, e.g., by InChIKey; if a large fraction of test candidates appear in the regularization pool, the rank@1 improvements over the no-regularization baseline would be suspect. A complementary falsifier would be retraining with regularization candidates restricted to molecules whose Tanimoto similarity to every test target is low and checking whether rank@1 gains persist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CMC contrastive multiview coding method and InfoNCE loss that JESTR adopts for its joint embedding training."},{"cited_title":"J., and Coley, C","cited_arxiv_id":null,"evidence_quote":"MIST, the spec-to-FP baseline, and also the source of the NPLIB1 data split used for one evaluation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CFM-ID, the pretrained spectral-prediction tool JESTR is compared to on held-out MassSpecGym subsets."},{"cited_title":"A., Thiessen, P","cited_arxiv_id":null,"evidence_quote":"PubChem, the molecular database from which regularization candidates and test candidate sets are retrieved by formula matching."}],"review_version":1}