{"id":"208e512d-2507-4ae5-bd9c-4014f313241c","arxiv_id":"2411.14079","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Using Recursive Feature Machines with a custom hybrid fingerprint yields lower solubility prediction errors than graph neural networks on ESOL and FreeSolv, while also producing feature-importance scores.","lead":"This paper applies a kernel-based machine learning method, Recursive Feature Machines, to predict molecular solubility across nine datasets and three molecular fingerprint types. It claims the method beats graph neural networks and identifies which molecular fragments drive solubility, which could support drug design.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HF construction is specified only as a 'correlation-based screening threshold' in §4.3.3, with no statement that the threshold is nested inside cross-validation; if it uses full-data labels, the reported ESOL 0.24 / FreeSolv 1.12 RMSE values are leakage-inflated.","rationale":"The reader's weakest-assumption analysis correctly identified HF feature selection as the point where the central claim is least secure. I agree, with one specification: the paper does not even define the screening statistic or its scope, so the manuscript cannot rule out label leakage, and the unusually large improvement over MACCS on ESOL is a red flag rather than evidence. I considered the unmatched Table 15 baselines as an alternative candidate; it is also serious, but it is secondary because it only affects the external GNN comparison, whereas leakage affects the paper's own RFM-HF results across all nine datasets. If the HF threshold is target-free, the verdict could soften to CONDITIONAL after a matched-split rerun; as written, with no code and no protocol, REJECT remains appropriate. I do not see deliberate misrepresentation; the issue is an evaluation protocol that does not support the reported numbers.","tokens_in":19213,"tokens_out":5565,"duration_ms":56734,"concrete_test":"Obtain or reimplement the HF construction and rerun Tables 11-14 with the correlation-screening threshold computed only inside each training fold (using training-fold descriptors/fragments and, if the paper's rule uses labels, training-fold labels only), then evaluate on held-out folds. If ESOL RFM-HF RMSE rises from 0.24 to approximately the MACCS value (0.67) or higher, leakage is confirmed and the SOTA claim is invalid; if it stays near 0.24, the leakage objection is refuted, though Table 15 still needs matched splits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that RFM-HF achieves SOTA predictive accuracy on solubility benchmarks, with ESOL RMSE 0.24 and FreeSolv RMSE 1.12 (Tables 14-15). All HF results depend on a custom multi-scale fingerprint, yet the only construction detail given is in §4.3.3: HF 'provides more flexible and enriched substructural information by applying a correlation-based screening threshold.' No equation, threshold value, or placement relative to the train/test split is given, and no code is released. If this screening uses correlation with the solubility label on the full dataset, the feature space is chosen with test information, and every HF test RMSE in Tables 11-14 is optimistically biased; the ESOL gap between RFM-HF (0.24) and RFM-MACCS (0.67) is exactly the pattern leakage would produce on a small 1,128-molecule dataset with a compact 311-bit fingerprint. Because Table 15 imports GNN results from other papers without a matched-split statement, the comparison to 'advanced graph neural networks' is independently unverifiable. The leakage question is the load-bearing issue: it determines whether RFM-HF is SOTA or an artifact of feature selection. The feature-importance overlap with PI/SHAP does not mitigate this, since those scores also come from the same HF pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript applies Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP) feature learning to Quantitative Structure-Property Relationship (QSPR) modeling for solubility prediction. It introduces a custom multi-scale hybrid fingerprint (HF), compares RFM against traditional ML and deep learning baselines on nine datasets, and claims state-of-the-art results on ESOL and FreeSolv, alongside local and global feature importance analysis. The paper also extends AGOP to Matern, Gaussian, and Rational Quadratic kernels and examines overfitting through kernel spectral properties.","tokens_in":19471,"tokens_out":3830,"duration_ms":41026,"significance":"If the central empirical claims are sound, the paper would be a useful contribution to interpretable QSPR modeling: applying RFM to molecular property prediction is a sensible transfer of recent kernel-feature-learning ideas, the nine-dataset comparison is broad, and the feature importance analysis is a relevant practical extension. The paper also gives explicit credit to the underlying RFM/AGOP theory and presents a concrete interpretability scheme. However, the headline SOTA claim is currently not adequately supported because the HF fingerprint construction is under-specified and a potentially leakage-prone feature screening step is not described relative to the train/test split; the comparison with graph neural networks in Table 15 is also not based on matched evaluation conditions. These issues are fixable but load-bearing.","major_comments":[{"comment":"The HF fingerprint is described only as applying \"a correlation-based screening threshold\" to select 68 descriptors and 243 fragment features, with no statement about whether this threshold is computed inside each cross-validation fold or on the full dataset. If the correlation screening uses the target variable on the full dataset, the reported test RMSE values, including ESOL RMSE 0.24 and FreeSolv RMSE 1.12, are optimistically biased by label leakage. This is the central claim of the paper, so the authors must specify the threshold value, the exact screening protocol, and demonstrate that the screening is nested within the resampling procedure, or re-report the results with leakage-free feature selection.","section":"§4.3.3, Tables 11–15"},{"comment":"The SOTA comparison for ESOL and FreeSolv imports MGCN, SchNet, GCN, GIN, DMPNN, ChemBFN, SPMM, ChemRL-GEM, and Uni-Mol results from their original publications, without establishing that the same train/test splits, preprocessing, and evaluation protocol were used. The RFM-HF column reports a single point estimate with no standard deviation or repeated-seed variation, whereas several imported baselines report means with standard deviations. The claimed advantage over \"advanced graph neural networks\" is therefore not verifiable as a head-to-head benchmark; the authors should either run the baselines under identical splits or clearly state the protocol differences and report variance for their own method.","section":"Table 15"},{"comment":"The local feature importance formula is not defined at the resolution needed for the claimed signed per-feature analysis. Equation (10), score(x_i) = x_i^T M x_i, is a scalar for each sample, and because M is an AGOP matrix (a sum of outer products of gradients) it is positive semidefinite, so the scalar quadratic form is nonnegative. The paper's discussion in §4.3.1 of features contributing \"positively\" or \"negatively\" to solubility for compounds 29 and 62 (Figure 8) cannot be derived from Eq. (10) unless a different per-feature score is defined. The authors should either specify the per-feature signed score, or soften the local interpretability claims to what Eq. (10) actually supports.","section":"§3.2.3, Eq. (10) and §4.3.1"}],"minor_comments":[{"comment":"Reference [45] duplicates reference [27] (Radhakrishnan et al., Science 2024); the duplicate should be removed and the in-text citation renumbered.","section":"References"},{"comment":"The caption says \"Root mean squared error (MSE) results,\" but the column is RMSE; the acronym should be corrected.","section":"Table 10"},{"comment":"The top-10 overlap between RFM and PI/SHAP on FreeSolv is only 3/10 and 5/10 in Table 6, which does not strongly support the wording \"high degree of correlation\"; the authors should report a null expectation or statistical significance for these overlaps.","section":"Tables 4–7"},{"comment":"No train/test split ratio, random seed, or repeated-evaluation protocol is stated for the RFM and baseline results; adding these details is essential for reproducibility.","section":"§4.2 / Tables 9–14"},{"comment":"The overfitting comparison uses \"consistent regularization parameters\" across kernels, but the actual values of the regularization parameter and kernel bandwidth are not given, making it hard to judge whether the spectral-decay comparison is confounded by hyperparameter choice.","section":"§4.3.2"},{"comment":"The text refers to compounds \"26\" and \"29\" when discussing the figure, while the preceding paragraph identifies the two examples as compounds \"29\" and \"62\"; this inconsistency should be fixed.","section":"§4.3.1, Figure 8"},{"comment":"There is a typo in the MACCS row for Arash (\"167bist\" instead of \"167bits\").","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful core idea, but the HF fingerprint is the linchpin of the SOTA claim and its construction is underspecified in exactly the place where leakage would be most damaging. I would encourage the editor to ask for a revised version that either proves the screening is inside the CV loop or reruns the benchmarks without leakage, and that makes the HF pipeline and splits fully reproducible. Releasing code would materially increase confidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this paper applies Recursive Feature Machines to solubility prediction and claims state-of-the-art results, but the central numbers are not trustworthy as reported. The hybrid fingerprint (HF) is built using a “correlation-based screening threshold” with no statement that the selection is nested inside cross-validation. If that threshold uses the full dataset’s labels, the ESOL RMSE of 0.24 is a leakage artifact, not a real improvement.\n\nWhat’s genuinely new: this is the first application of RFM to QSPR, the extension to Matern, Gaussian, and rational quadratic kernels is reasonable, and the feature importance analysis with PI/SHAP comparison is a nice practical touch. The kernel-overfitting spectral analysis is also suggestive. Those parts are worth a look.\n\nThe soft spots are concentrated where the claims are biggest. The HF construction is underspecified—no threshold value, no equation, no code. The comparison to GNNs in Table 15 imports numbers from other papers without confirming matched train/test splits. And on the small ESOL dataset, the gap between RFM-HF (0.24) and RFM-MACCS (0.67) is exactly the signature of target leakage in feature selection. The paper’s own description doesn’t rule that out. That’s not a minor omission; it’s the load-bearing wall.\n\nI’m not saying the method is fake. RFM is a real technique with published theory, and the kernel extension work is probably salvageable. But the main claim—that RFM-HF surpasses GNNs—needs to be redone with the feature selection anchored inside the cross-validation loop, and the code and splits made available. Until then, the SOTA claim should be treated as unverified.\n\nWho’s this for? Someone working on interpretable QSPR might find the kernel comparison and feature-scoring analysis worth reading. But anyone citing the ESOL/FreeSolv numbers should wait.\n\nMy recommendation: send it to peer review, but with a strong request for code and explicit leakage checks. A good referee can sort out whether the HF screening is nested. If the authors can show that, the paper has real value. If they can’t, the core result collapses. Either way, it deserves scrutiny rather than a desk reject.","headline":"The SOTA solubility claims rest on an underspecified correlation-based feature selection that may leak test labels, so the headline numbers are unverified, though the kernel extensions and interpretability analysis are worth a second look.","tokens_in":20053,"tokens_out":3189,"would_cite":false,"duration_ms":29439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A kernel machine that learns its own feature weights achieves state-of-the-art molecular solubility prediction and provides chemically interpretable feature importance.","keywords":["recursive feature machines","average gradient outer product","kernel ridge regression","molecular solubility prediction","multi-scale hybrid fingerprints","feature importance","QSPR","interpretable machine learning"],"falsifier":"Re-run RFM-HF on ESOL and FreeSolv with the HF feature screening performed only on the training folds of the same splits and compare the RMSE to the reported 0.24 and 1.12; if the errors increase substantially or the gap to MACCS/Morgan narrows, the SOTA claim is an artifact of leakage.","tokens_in":18961,"feed_emoji":"🧪","tokens_out":8542,"duration_ms":72875,"temperature":0.7,"pith_summary":"This paper claims that Recursive Feature Machines (RFM) — a fast kernel-ridge model that learns a feature-weighting matrix through the average gradient outer product — can match or beat deep graph neural networks at predicting molecular solubility, while also exposing which substructures drive the prediction. The authors pair RFM with a custom multi-scale hybrid fingerprint (HF) that concatenates global physicochemical descriptors with SMILES-based fragment counts, and report state-of-the-art errors on the ESOL and FreeSolv benchmarks (RMSE 0.24 and 1.12) across nine solubility datasets. They also show that the same learned matrix yields local and global feature-importance scores whose top features overlap substantially with SHAP and permutation importance, and that removing redundant HF features does not degrade accuracy. If these results hold, kernel-based QSPR models become an attractive combination of accuracy, speed, and interpretability for drug-discovery screening.","feed_headline":"Kernel model tops GNNs at solubility prediction and says why","feed_subtitle":"RFM-HF reaches RMSE 0.24 on ESOL, and its feature scores trace which molecular substructures matter.","key_machinery":"The AGOP matrix M, recursively updated as the average gradient outer product of the kernel predictor's output with respect to its inputs, is the load-bearing object. It enters the kernel as $K(x,x') = \\exp(-\\gamma \\sqrt{(x-x')^T M (x-x')})$ for the Laplace case, with analogous modifications for Matern, Gaussian, and rational-quadratic kernels, so that each iteration re-weights features before re-solving kernel ridge regression. The same M produces local importance scores $x_i^T M x_i$ and, averaged over the dataset, global rankings. The companion piece is the multi-scale hybrid fingerprint (HF): 68 global descriptors plus 243 SMILES-fragment counts (41+139 for three smaller Samuel subsets), constructed by SMILES Pair Encoding and functional-group fragmentation, which gives each HF bit a direct chemical meaning and avoids Morgan fingerprint hash collisions.","core_discovery":"The central discovery is that a kernel machine can acquire deep feature learning through AGOP: in each iteration, RFM solves kernel ridge regression and then updates a matrix M equal to the average outer product of the prediction gradient with respect to the input features; this M is folded into the kernel as a Mahalanobis distance. The paper applies this machinery to molecular solubility prediction for the first time, defining a hybrid fingerprint whose bits are directly interpretable substructures, and obtains results that surpass the reported errors of graph neural networks and graph transformers on ESOL and FreeSolv. The same M matrix doubles as an interpreter: the local score $x_i^T M x_i$ for a molecule and the average over molecules reproduce the global feature rankings that SHAP and permutation importance give, in a fraction of the time. The authors further show that AGOP generalizes beyond the Laplace kernel to Matern, Gaussian, and rational-quadratic kernels, with Matern and Laplace giving the best generalization, and they relate each kernel's overfitting tendency to the eigenvalue spectrum of the learned M matrix.","pith_inferences":["A direct consequence the authors do not state: if the correlation-based HF feature screening is not nested inside the cross-validation loop, the ESOL and FreeSolv RMSE numbers (0.24 and 1.12) are likely optimistic; re-running with screening inside each training fold would settle this without changing the method.","The AGOP importance scores could be aggregated across datasets to derive transferable substructure-solubility rules, e.g., fragments that consistently increase solubility, which would give medicinal chemists testable design hypotheses.","The same hybrid fingerprint and RFM pipeline could be ported to other endpoints, such as permeability, toxicity, or logP, where bit-level interpretability is equally valuable.","The eigenspectrum-based overfitting diagnosis used here for kernels might also serve as a practical diagnostic for choosing between molecular representations, not just kernels."],"forward_implications":["If RFM-HF's reported accuracy is reproducible, kernel ridge regression with AGOP becomes a direct competitor to GNNs and graph transformers for molecular property prediction, at much lower computational cost.","The $x_i^T M x_i$ score gives chemists per-molecule, per-substructure attributions without post-hoc explanation methods, so feature importance and prediction come from the same calculation.","The redundancy-filtering result implies the HF fingerprint can be aggressively compressed without losing accuracy, which simplifies deployment and interpretation.","The kernel comparison suggests AGOP is not Laplace-specific; Matern and Laplace kernels both carry the benefit, and spectral analysis of M can be used to pick among kernels.","On the nine solubility datasets, HF representation improves almost every model tried, not just RFM, so hybrid descriptor-fragment fingerprints are a broadly useful representation."],"supporting_citations":[{"why":"Supplies the AGOP and RFM formulation that the paper applies to QSPR.","marker":"[26]-[27]"},{"why":"AqSolDB, the largest aqueous solubility dataset used for training and evaluation.","marker":"[35]"},{"why":"ESOL, the benchmark dataset where RFM-HF reports its best RMSE of 0.24.","marker":"[37]"},{"why":"FreeSolv, the hydration free-energy benchmark where RFM-HF reports RMSE 1.12.","marker":"[38]"},{"why":"Samuel datasets for organic solvents and water, five of the nine evaluation sets.","marker":"[39]"},{"why":"GINE set-representation learning baseline that RFM-HF is compared against.","marker":"[54]"},{"why":"ChemBFN, a SOTA baseline whose reported ESOL RMSE RFM-HF claims to beat.","marker":"[55]"},{"why":"Uni-Mol, a 3D pre-trained foundation model baseline used in the SOTA comparison.","marker":"[58]"},{"why":"SHAP, the feature-attribution method used to validate RFM's global importance rankings.","marker":"[24]-[25]"},{"why":"Permutation importance, the second attribution method used to validate RFM's rankings.","marker":"[33]"}],"fun_headline_variants":["Kernel model with AGOP beats GNNs and explains why","RFM-HF: interpretable kernel model tops GNNs on solubility","Kernel model learns deep features, beats GNNs on solubility","Hybrid fingerprint boosts kernel model past GNNs on solubility"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hybrid fingerprint is built by a correlation-based screening threshold that may use the target variable on the full dataset; if that selection is not nested inside cross-validation, the reported test RMSE values are inflated by label leakage.","fun_headline_variants_meta":{"raw":{"variants":["Kernel model with AGOP beats GNNs and explains why","RFM-HF: interpretable kernel model tops GNNs on solubility","Kernel model learns deep features, beats GNNs on solubility","Hybrid fingerprint boosts kernel model past GNNs on solubility"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00137,"raw_usage":{"total_tokens":5598,"prompt_tokens":1035,"completion_tokens":4563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":4487}},"tokens_in":651,"tokens_out":4563,"duration_ms":28353,"temperature":1.0,"reasoning_tokens":4487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:33:16.376243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run RFM-HF on ESOL and FreeSolv with the HF feature screening performed only on the training folds of the same splits and compare the RMSE to the reported 0.24 and 1.12; if the errors increase substantially or the gap to MACCS/Morgan narrows, the SOTA claim is an artifact of leakage.","supporting_citations":[{"cited_title":"A Bayesian Flow Network Framework for Chemistry Tasks","cited_arxiv_id":"2407.20294","evidence_quote":"ChemBFN, a SOTA baseline whose reported ESOL RMSE RFM-HF claims to beat."},{"cited_title":"Uni-mol: A universal 3d molecular representation learning framework","cited_arxiv_id":null,"evidence_quote":"Uni-Mol, a 3D pre-trained foundation model baseline used in the SOTA comparison."},{"cited_title":"Permutation importance: a corrected feature importance measure[J]","cited_arxiv_id":null,"evidence_quote":"Permutation importance, the second attribution method used to validate RFM's rankings."},{"cited_title":"Machine learning with physicochemical relationships: solubility prediction in organic solvents and water","cited_arxiv_id":null,"evidence_quote":"Samuel datasets for organic solvents and water, five of the nine evaluation sets."},{"cited_title":"Molecular set representation learning","cited_arxiv_id":null,"evidence_quote":"GINE set-representation learning baseline that RFM-HF is compared against."},{"cited_title":"AqSolDB, a curated reference set of aqueous solubility and 2D descriptors for a diverse set of compounds","cited_arxiv_id":null,"evidence_quote":"AqSolDB, the largest aqueous solubility dataset used for training and evaluation."},{"cited_title":"ESOL: estimating aqueous solubility directly from molecular structure","cited_arxiv_id":null,"evidence_quote":"ESOL, the benchmark dataset where RFM-HF reports its best RMSE of 0.24."},{"cited_title":"FreeSolv: a database of experimental and calculated hydration free energies, with input files","cited_arxiv_id":null,"evidence_quote":"FreeSolv, the hydration free-energy benchmark where RFM-HF reports RMSE 1.12."}],"review_version":1}