{"id":"2f6b8a00-41f5-4264-b1f5-3cbab1506f2e","arxiv_id":"2501.18439","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"MolGraph-xLSTM applies xLSTM to atom- and motif-level molecular graphs with a multi-head mixture-of-experts and reports benchmark gains on 10 property prediction datasets.","lead":"This paper presents MolGraph-xLSTM, a machine learning model that predicts molecular properties by combining atom-level and motif-level graph representations with an extended LSTM (xLSTM) and a mixture-of-experts layer. The authors claim consistent improvements over five baselines across ten drug-discovery benchmarks, though the size of the claimed gains is not fully supported by the reported tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline fidelity and the averaging protocol are load-bearing: the abstract's 3.18%/3.83% averages do not reproduce from Tables 1-2, and TransFoxMol's ESOL RMSE of 0.930 is far worse than its original ~0.62, so the claimed consistent gains are not established.","rationale":"The reader's verdict of REJECT is well supported, and the weakest assumption they identified is the same one I consider load-bearing: the five baselines must be implemented and tuned faithfully for any relative improvement to be meaningful. My independent check of Tables 1 and 2 strengthens this concern: the abstract's average improvements do not match the table under natural averaging rules, the BBBP text misidentifies the best baseline, and the model actually loses to FP-GNN on Caco2. The TransFoxMol ESOL value of 0.930 is particularly damning because it is far outside the range reported in the original TransFoxMol publication, suggesting an underpowered baseline run rather than a genuinely stronger model. This is not an attack on the architecture itself; the dual-level graph representation and MHMoE are plausible, and the ablation study is informative. But the paper's central claim is a benchmarking claim, and benchmark claims stand or fall on the fidelity of the comparisons. If the official TransFoxMol code reproduces the paper's 0.930 ESOL RMSE under the stated protocol, then my concern would be mitigated; if it reproduces the original ~0.62, the central claim loses its empirical basis. The paper could become acceptable only after faithful baseline implementations, a clearly specified and reproducible averaging procedure, and validation-only hyperparameter selection are documented. As presented, the evidence does not support the claimed consistent improvements, so the REJECT verdict should stand.","tokens_in":17962,"tokens_out":8117,"duration_ms":65133,"concrete_test":"Run the official TransFoxMol implementation on ESOL and BBBP using the exact 8:1:1 random split, validation-based early stopping, and hyperparameters specified in this paper (or, if that protocol is unavailable, the original paper's protocol), and compare the resulting RMSE and AUROC to Tables 1-2 here and to the published TransFoxMol values. If the official ESOL RMSE is near 0.62 rather than 0.930, or the BBBP AUROC is near the original publication's value rather than 0.881, the baseline is underpowered and the claimed 7.54% ESOL and 7.03% BBBP improvements are artifacts of the comparison. Additionally, recompute the abstract's average improvements from Tables 1-2 under every averaging rule consistent with the text; if no rule reproduces 3.18% AUROC and 3.83% RMSE, the headline numbers are unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical superiority over five baselines, so the comparison must rest on faithful baseline runs and an explicit, correct averaging rule. Both are insecure. First, the abstract's headline numbers are not reproducible from the paper's own tables. Comparing MolGraph-xLSTM to the best baseline per dataset, the average relative AUROC improvement is (5.45+0.59+1.01+7.03+2.12+0.78)/6 = 2.83%, not 3.18%; the average relative RMSE reduction is (7.54+0.54+4.12-2.44)/4 = 2.44%, not 3.83%. Comparing to the mean of all baselines gives 5.25% and 11.54%, respectively. No stated rule yields the abstract's numbers. Second, the baseline comparison itself is suspect: TransFoxMol's ESOL RMSE is 0.930 ± 0.261, whereas its original publication reports roughly 0.62; on BBBP the paper's Table 1 text names TransFoxMol as the best baseline with 0.896, but the table shows DMPNN at 0.896 and TransFoxMol at 0.881. Third, Table 2 shows MolGraph-xLSTM (RMSE 0.503) is worse than FP-GNN (0.491) on Caco2, contradicting the claim of consistent performance across all datasets. Together these undermine the central claim that the model is a consistently better predictor than the five named baselines, independent of the architectural plausibility of the GNN-xLSTM and MHMoE design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MolGraph-xLSTM, a molecular property prediction model that combines atom-level and motif-level graph representations. The atom-level branch uses a GCN with jumping-knowledge connections followed by an xLSTM layer, while the motif-level branch applies xLSTM directly to a simplified motif graph; the two branches are merged and refined by a multi-head mixture-of-experts module before property prediction. The authors evaluate the model on six classification and four regression datasets, comparing against FP-GNN, DeeperGCN, DMPNN, HiGNN, and TransFoxMol, and report AUROC/AUPRC and RMSE/PCC metrics. They also present ablations, a node-ordering robustness study, hyperparameter analyses, and an interpretability case study. The abstract claims average AUROC improvements of 3.18% and RMSE reductions of 3.83% over baselines, with per-dataset improvements up to 7.03% on BBBP and 7.54% on ESOL.","tokens_in":18256,"tokens_out":6917,"duration_ms":57422,"significance":"If the empirical claims were fully supported, the paper would offer a plausible and useful combination of known components: GNN message passing for local structure, xLSTM for long-range dependencies at both atom and motif levels, and MHMoE for representation refinement. The manuscript provides code, ablation studies, and an interpretability analysis, which are valuable assets. However, the central claim of consistent, roughly 3--4% average gains over five baselines is not currently established: the headline averages do not reproduce from the paper's own tables, the BBBP baseline attribution is internally inconsistent, the Caco2 result contradicts the 'consistent performance' claim, and the reported TransFoxMol baseline values are far outside the range reported in its original publication. Because the central contribution is empirical benchmarking superiority, these issues are load-bearing and require correction before the paper's conclusions can be accepted.","major_comments":[{"comment":"The headline averages do not reproduce from the paper's own tables under any stated rule. Computing relative improvement over the best baseline per dataset from the AUROC column of Table 1 gives (5.45 + 0.59 + 1.35 + 7.03 + 2.11 + 0.78)/6 ≈ 2.88%, not 3.18%; for RMSE in Table 2 it gives (7.54 + 0.54 + 4.12 − 2.44)/4 ≈ 2.44%, not 3.83%. Computing against the mean of all five baselines gives approximately 5.23% and 11.54%, respectively. Since no averaging convention is stated in the paper, the abstract's 3.18% and 3.83% figures are unsupported and must be corrected or removed.","section":"Abstract and §4.3, Tables 1–2"},{"comment":"The text reporting the BBBP result is internally inconsistent: it states that the best baseline is TransFoxMol at 0.896 ± 0.024, whereas Table 1 lists TransFoxMol at 0.881 ± 0.015 and DMPNN at 0.896 ± 0.014. The claimed 7.03% improvement therefore cannot be attributed to the stated baseline; the table and text need to be reconciled and the comparison baseline clearly identified.","section":"§4.3, Table 1"},{"comment":"Baseline fidelity is load-bearing for the central empirical claim, and the reported baseline values are not credible in at least one case: TransFoxMol's ESOL RMSE of 0.930 ± 0.261 is far worse than the value reported in the original TransFoxMol publication, and its BBBP AUROC of 0.881 is lower than several other baselines in the same table. Because no implementation details, hyperparameters, or seed controls are provided for the baselines, the reader cannot determine whether these are faithful baseline runs; the authors should provide this information or rerun the baselines with public implementations.","section":"§4.2, Tables 1–2"},{"comment":"On Caco2, MolGraph-xLSTM has RMSE 0.503 ± 0.004, which is worse than FP-GNN's 0.491 ± 0.023. This directly contradicts the abstract's claim of 'consistent performance across all datasets' and the averaged RMSE reduction claim, and the result is not discussed in Section 4.3. The claims need to be qualified to per-dataset results, or the contradiction needs to be explained.","section":"Table 2 and Abstract"}],"minor_comments":[{"comment":"Equation (17) has unbalanced parentheses and a malformed gating term: it should read f_s^{MoE} = Σ_{e=1}^n G(f_s)_e E_e(f_s), with the expert index matching the gating output.","section":"§3.4, Eq. (17)"},{"comment":"The feature tables are referenced only as 'Table ??' three times in Section 3.1; these cross-references need to be resolved to the actual table numbers.","section":"§3.1"},{"comment":"Equation (11) contains a double plus sign in 'hl+1_p = ReLU(LN(hl_p)) + +vnl+1', which appears to be a typographical error.","section":"§3.2.1, Eq. (11)"},{"comment":"The headings 'Varing Number of Experts and Heads' and 'Varing Number of Jump Layers' contain a typo; 'Varing' should be 'Varying'.","section":"§4.6"},{"comment":"The text references panels (e) and (f) of Figure 6, but the figure caption does not label individual panels, making the references difficult to follow.","section":"§4.5, Figure 6"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is a genuine new combination: atom-level and motif-level graphs processed by GNN plus xLSTM, with multi-head mixture-of-experts and a supervised contrastive loss. I don't see this exact stack in the cited literature, and the ablation study gives some honest evidence that both graph levels and the MHMoE module contribute. Code is on GitHub, which is a real plus.\n\nThat said, the empirical claims as stated don't survive contact with the tables. The abstract promises an average AUROC improvement of 3.18% and RMSE reduction of 3.83%, but no reasonable averaging rule over Tables 1 and 2 reproduces those numbers. For example, versus the best baseline per dataset the RMSE reduction averages about 2.4%, not 3.83%.\n\nThe baseline fidelity is the bigger worry. TransFoxMol's ESOL RMSE is reported at 0.930 ± 0.261, while its original paper lands near 0.62; that is more than a standard deviation off. On BBBP the text names TransFoxMol as the best baseline at 0.896, but the table has DMPNN at 0.896 and TransFoxMol at 0.881. And on Caco2 the model is worse than FP-GNN, which undercuts the 'consistent performance across all datasets' claim. These are not cosmetic blemishes; they are load-bearing for a paper whose main contribution is empirical superiority.\n\nI also note the regression tasks use random splits, and the hyperparameters are per-dataset with no demonstrated guarantee that selection was validation-only. Either of those alone would be a minor concern; together with the baseline issues they make the headline results unreliable as a comparison.\n\nI want to be fair: the model design is reasonable, the ablations are informative, and the interpretability analysis is a nice touch, even if not rigorously benchmarked. The paper is not sloppy in its writing or citation pattern. But the central claim of consistent, sizeable gains over five named baselines is not established by the evidence presented. A thorough revision with faithful baseline implementations, scaffold splits for the small regression sets, a clear averaging rule, and validation-only hyperparameter selection could make this a solid incremental contribution. As submitted, I would not trust the numbers.\n\nFor peer review: yes, I'd send it to a knowledgeable referee, but with a strong request to focus on baseline fidelity and split protocol. The architecture is worth evaluating; the current evidence is not.","headline":"A plausible new architecture, but the headline performance claims don't reproduce from the paper's own tables and the baseline runs look underpowered.","tokens_in":18874,"tokens_out":2393,"would_cite":false,"duration_ms":22662,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-based xLSTM that reads each molecule at atom and motif scales reports consistent predictive gains over five baselines across ten property-prediction datasets.","keywords":["molecular property prediction","molecular graph representation learning","multi-head mixture-of-experts","xLSTM","drug discovery","long-range dependencies","graph neural networks"],"falsifier":"Re-run MolGraph-xLSTM and the five baselines on the same ten splits using each baseline's official implementation and recommended hyperparameters, then check whether MolGraph-xLSTM still beats them; in particular, if TransFoxMol's ESOL RMSE of 0.930 and BBBP AUROC of 0.881 move closer to the values in its original publication, the reported 3.18% AUROC and 3.83% RMSE gaps would shrink or disappear.","tokens_in":17699,"feed_emoji":"🧪","tokens_out":6693,"duration_ms":56265,"temperature":0.7,"pith_summary":"The paper argues that GNNs' limited ability to capture long-range molecular dependencies can be remedied by inserting xLSTM into the representation pipeline at two structural scales. MolGraph-xLSTM processes an atom-level graph with a GNN plus jumping knowledge followed by xLSTM, and separately processes a motif-level graph with xLSTM, then fuses the pooled embeddings through a multi-head mixture-of-experts. Across ten classification and regression datasets, the authors report an average AUROC gain of 3.18% and an average RMSE reduction of 3.83% over five baselines, with the largest single-dataset improvements being 7.03% on BBBP and 7.54% on ESOL. If the comparisons are faithful, this offers drug-discovery pipelines a direct way to add long-range modeling to molecular graphs while preserving local and substructural detail.","feed_headline":"Dual-scale xLSTM bests five baselines across ten molecule datasets","feed_subtitle":"Average AUROC improves 3.18% and RMSE falls 3.83%, with the biggest single-task gains on BBBP and ESOL.","key_machinery":"The central objects are the paired molecular graphs: the atom-level graph $G_{atom}$ and the motif-level graph $G_{motif}$, built by decomposing the atom graph into rings, non-cyclic functional groups, and carbon-carbon single bonds. The carrying mechanism is the insertion of xLSTM at both levels: after a GCN with residual connections and virtual nodes produces per-atom features, jumping knowledge concatenates layer outputs, and xLSTM treats those node features as a sequence of length $N_{atom}$; the smaller motif graph, which is closer to sequential data, goes straight into xLSTM. A multi-head mixture-of-experts then splits the summed pooled features into $h$ segments and routes each segment through the top-$K$ experts, and a supervised contrastive loss on atom, motif, and fused embeddings pulls same-label molecules closer together. The paper's claimed gains come from this combination rather than from any single component, as the ablations on Sider and FreeSolv show.","core_discovery":"On the paper's own terms, the central claim is that molecular property prediction improves when a molecule's representation is computed from both atoms and chemically meaningful substructures, with xLSTM applied at both levels so that information from distant parts of the molecule is not squashed away by repeated message passing. The atom-level branch uses a GCN with residual connections and virtual nodes, concatenates all layer outputs through jumping knowledge, and then treats the node set as a sequence for xLSTM; the motif-level branch feeds substructure nodes directly into xLSTM. The three pooled feature vectors (GNN, atom-xLSTM, motif-xLSTM) are summed and refined by a multi-head mixture-of-experts that splits the vector into segments and routes each segment to its top experts. The paper reports the best or second-best score on most of the ten benchmarks and interprets the highest-weighted motifs and atoms as biologically meaningful, such as sulfonamide and carboxylic-acid substructures.","pith_inferences":["A direct test of the long-range-dependency claim would compare MolGraph-xLSTM against the same atom-level GNN without xLSTM, or against a Transformer-based graph model, holding parameter count and training budget fixed; the paper's ablations remove the motif branch and MHMoE but do not isolate xLSTM itself.","The node-order robustness result suggests xLSTM is not exploiting a canonical chemical ordering, so a permutation-invariant variant that averages over several depth-first-search orders could stabilize regression predictions without relying on RDKit's default ordering.","Because the interpretability analysis links high-weight substructures to known toxicophores, the framework could be used prospectively to screen candidates for specific side effects before synthesis, with the model's highlighted motifs serving as testable hypotheses.","The dual-scale xLSTM design is not chemistry-specific beyond graph construction and could transfer to other structured prediction problems with long-range dependencies, such as protein or materials property prediction, where substructure-like groupings can be defined."],"forward_implications":["If the reported gains hold under faithful baseline tuning, adding xLSTM to molecular graphs becomes a low-cost upgrade over GNN-only models for ADMET-style classification tasks.","The dual-scale design gives an interpretability handle, since motifs and atoms with the highest xLSTM weights can be checked against known toxicophores and used to flag risky substructures before synthesis.","The framework extends to other graph-level prediction problems where long-range dependencies matter, such as drug-target interaction or materials property prediction, because it does not depend on chemistry-specific sequence encodings.","Because the motif-level branch is noticeably weaker on regression, enriching substructure features with geometric or electronic descriptors would likely improve continuous-target predictions within the same architecture."],"supporting_citations":[{"why":"Supplies the xLSTM architecture (sLSTM and mLSTM blocks) that the paper adapts to molecular graphs for long-range dependency modeling.","marker":"[19]"},{"why":"Provides ReLMole, the method used to decompose atom-level graphs into motif-level graphs with rings, functional groups, and C-C single bonds.","marker":"[31]"},{"why":"Defines the multi-head mixture-of-experts module used to refine the fused atom- and motif-level embeddings.","marker":"[24]"},{"why":"Supplies jumping knowledge, which concatenates GNN layer outputs before xLSTM in the atom-level branch.","marker":"[23]"},{"why":"Contributes the MoleculeNet benchmark datasets and evaluation protocol used for eight of the ten property-prediction tasks.","marker":"[33]"},{"why":"Is the DMPNN baseline, a bond-centered message-passing model the paper compares against.","marker":"[6]"},{"why":"Is the DeeperGCN baseline and the source of the residual and virtual-node techniques used in the atom-level GNN.","marker":"[7]"},{"why":"Is the TransFoxMol GNN-plus-transformer baseline whose reported scores are the weakest comparison point.","marker":"[12]"},{"why":"Is the HiGNN baseline that also uses motif and fragment-level graphs, making it the closest structural comparison.","marker":"[26]"}],"fun_headline_variants":["Dual-scale xLSTM with MoE lifts molecule prediction up to 7.5%","Atom and motif xLSTM boosts accuracy on 10 molecular benchmarks","Graph xLSTM with mixture-of-experts tops baselines in drug discovery","Two-level xLSTM plus MoE: up to 7.5% gain on molecule tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The performance gains are measured against five baselines whose reported scores are taken as faithful; if those baselines were run with suboptimal settings or hyperparameters, the claimed average improvements would be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Dual-scale xLSTM with MoE lifts molecule prediction up to 7.5%","Atom and motif xLSTM boosts accuracy on 10 molecular benchmarks","Graph xLSTM with mixture-of-experts tops baselines in drug discovery","Two-level xLSTM plus MoE: up to 7.5% gain on molecule tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":3033,"prompt_tokens":1039,"completion_tokens":1994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1904}},"tokens_in":655,"tokens_out":1994,"duration_ms":13740,"temperature":1.0,"reasoning_tokens":1904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T23:29:17.620328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MolGraph-xLSTM and the five baselines on the same ten splits using each baseline's official implementation and recommended hyperparameters, then check whether MolGraph-xLSTM still beats them; in particular, if TransFoxMol's ESOL RMSE of 0.930 and BBBP AUROC of 0.881 move closer to the values in its original publication, the reported 3.18% AUROC and 3.83% RMSE gaps would shrink or disappear.","supporting_citations":[{"cited_title":"Journal of Chemical Information and Modeling 62(22), 5361–5372 (2022)","cited_arxiv_id":null,"evidence_quote":"Provides ReLMole, the method used to decompose atom-level graphs into motif-level graphs with rings, functional groups, and C-C single bonds."},{"cited_title":"In: ICML","cited_arxiv_id":null,"evidence_quote":"Supplies jumping knowledge, which concatenates GNN layer outputs before xLSTM in the atom-level branch."},{"cited_title":": Analyzing learned molecular repre- sentations for property prediction","cited_arxiv_id":null,"evidence_quote":"Is the DMPNN baseline, a bond-centered message-passing model the paper compares against."},{"cited_title":"Briefings in Bioinformatics 24(5), 306 (2023)","cited_arxiv_id":null,"evidence_quote":"Is the TransFoxMol GNN-plus-transformer baseline whose reported scores are the weakest comparison point."},{"cited_title":"Journal of Chemical Information and Modeling 63(1), 43–55 (2022) 24","cited_arxiv_id":null,"evidence_quote":"Is the HiGNN baseline that also uses motif and fragment-level graphs, making it the closest structural comparison."}],"review_version":1}